DOCOHEIT BESUHE 



ED 133 556 

TITLE 

INSTITUTION 
SPONS AGENCY 



REPORT NO 
PUB DATE 
CONTRACT 
NOTE 

EDRS PRICE 
DESCRIPTORS 



08 



CB 009 625 



IDENTIFIERS 



ABSTRACT 



CurriculuB for Graduate Program to Prepare Vocational 
Education Curriculum Specialists (TECS) • Module 6: 
Preparing for Curriculum Evaluation, 
Washington State Univ.^ Pullman, Coll, of 
Education. 

Bureau of Occupational and Adult Education (DHEW/OE) , 

Washington^ D-C, Div, of Research and 

Demonstration. 

VT-103-456 

[76] 

OEC-0-74-9287 

118p-; For related documents see CE 009 624-632 

MF-$0,83 HC-$6,01 Plus Postage. 
Curriculum Development; ♦Curriculum Evaluation; 
♦Evaluation Criteria; ♦Evaluation Methods; Graduate 
Study; Higher Education; Instructional Materials; 
Learning Activities; Learning Modules; Performance 
Based Education; Post Secondary Education; Secondary 
Education; Specialists; Teacher Education; ♦Teacher 
Education Curriculum; ♦Vocational Education 
Vocational Education Curriculum Specialists 



Part of an eight-;module, graduate level, 
competency-based curriculum to prepare specialists in vocational 
education curriculum, this module provides information and 
experiences to enable the curriculum specialist to coordinate 
evaluation in the development and implementation of vocational 
curriculums- The module is divided into two parts. Part 1 presents 
basic concepts and background knowledge of the evaluation process, 
including such topics as evaluation terminology, relationship of 
evaluation to the learning process, various evaluation models, 
criteria and design, and evaluation instruments. Part 2 is devoted to 
carrying out the evaluation process and field experience, involving 
the following activities: Assembling,^ analyzing, interpreting, and 
presenting evaluation data, evaluating program proposals and 
curriculum guides, and developing a program evaluation in an actual 
school setting. Organization oif the module is By major performance 
objectives followed by related learning activities, A pretest, 
posttest, and glossary of terms complete the modjlile, (NJ) 



♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦^^♦★♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦**** 

♦ Documents acquired by ERIC include many informal unpublished ♦ 

♦ materials nc 'ailable from other; sources- ERIC makes every effort ♦ 

♦ to obtain ti st copy available. Nevertheless, items of marginal ♦ 

♦ reproducibilx i are often encountered and this affects the quality ♦ 

♦ of the microfiche and hardcopy reproductions ERIC makes available ♦ 

♦ via the ERIC Document Reproduction Service (EDRS), EDRS is not ♦ 

♦ responsible for the quality of the original document. Reproductions ♦ 

♦ supplied by EDRS are the best that can be pade from the original. ♦ 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦*♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦*♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦*♦♦♦ 



m 

LTV 



CURRICULUM 
for 

GRADUATE PROGRAM 
to Prepare 

VOCATIONAL EDUCATION CURRICULUM SPECIALISTS 



VECS 

PffOJfCT 




Preparing for 

Curriculum 

Ei^aluaNon 



y MODULE B 



by the 
Department of Education 
Washington State University 
Pullman, Washington 99163 



I 

via 

I \ 

ERIC ^ 



Developed and disseminated pursuant to grant No. OEC 0-74-9287 
for the Curriculum Development Branch 
of the Division of Research and Demons tration» 
Bureau of Occupational and Adult Education 
U. S. Office of Education, Department of Health, Education, and Welfare 
under Part I - CurriculuiP Development in Vocational and Techiiical Education 
Vocational Education Amendments of 1968, Public Law 90-576 



3 



The content of this module was used as a part of a graduate course 
in Vocational Technical Education at Washington State University. The 
effectiveness of the curriculum material was evaluated on the basis of: 

1. Performance data from pre and post tests and other 
evaluative techniques used In the course. 

2. Use of a Curriculum Evaluation Questionnaire which 
was developed, administered and summarized by the 
third party evaluator, Northwest Regional Education 
Laboratory. This questionnaire provided a faculty 
and student rating of: Individualization of the 
curriculum, choice of learning settings and quality 
of curriculini. 

Revisions in final drafts utilized these evaluations* 
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THE VOCATIONAL EDUCATION CURRICULUM SPECIALIST 
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INTRODUCTION 
Purpose 

This module Is designed to provide the vocational education curriculum 
specialist with the necessary knowledge, awareness, and skills to coordinate 
systematic evaluation in the development and inifl'lementation of vocational 
curricula. 

Rationale 

The learning process involves: (1) delineation of goals and objectives, 
(2) the definition of learning activities, and (3) the assessment or evalua- 
tion of the degree to which the objectives are achieved. Evaluation provides 
the necessary feedback in the learning process to ascertain if appropriate 
progress is being made towards stated goals (formative evaluation) or the 
degree to which the goals have been reached - (summative eval.iat:ion) . Forma- 
tive evaluation enables the planner, through systematic feedba^, ^o know if 
the learner is "on course" or if the objectives need modification in content 
or level of expectation. Summative evaluation enables the planner to make 
overall Judgments and decisions about the success of the learning activities. 

Assumptions 

It can be assumed that: 

1. All learning involves evaluation or feedback, either formal (planned) 
or informal (incidental). 

2. The most successful type of evaluation will be planned systematically 
and related to desirable learner goals. 

3. The need for systematic evaluation is important because the curricultim 
specialist will be responsible for introducing new methods and materials 
and with restructuring existing resources. 



A. Decisions based on sound evaluation methods, with carefully and 
systematically collected data, will be far superior to those based 
on custom, tradition, and subjective Impressions. 

Overview of Module 
This module on evaluation will be divided Into two parts, each having 
somewhat unique types of learning activities. ; 
PART I 

Part I will Include basic concepts and background knowledge about the 
evaluation process. Content areas will Include the usual topics such as 
evaluation terminology, relationship of evaluation to the learning process, 
various evaluation models, criteria and design, evaluation Instrtnnents , etc. 
The learning activities will be traditional In nature Including broad exposure 
through reading, InSilvldual study, lecture discussion, and written assignments 

Concepts and knowledges gained from the first part of this module will 
enable the learner to attain the objectives for the "application ox^lented" 
Part II 

Performance Objectives 

A summary of specific performance objectives for Part 1 are as|follows; 

1. Define the terminology and explain the concepts In educatlo^l 
measurement and evaluation. \ 

2. Diagram and explain the relationship between evaluation and other 
cutrlcular concepts. 

3. Explaih the key salient features and usage of evaluation models 
having relevance for vocational education. 

4. Select and/or develop criteria that may be applied In evaluation 
of vocational educational curricula. 

5. Specify the sources of error and evaluate the criteria used In 
measures used to determine student outcomes In vocational programs. 
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Part II of this module la devoted to learning to carry out the evalua- 
tion process and field experience. Objectives 6 and 7 will Involve the 
appraisal of proposals and curriculum guides In the VTE area which, though 
they do not represent a .total curriculum, represent a somewhat complete « 
gestalt of learning activities and provide the learner practice In Implemen- 
ting evaluative criteria In an actual program context* 

A variety of curriculum programs, course plans, and project proposals 
may be supplied by the Instructor for review. The evaluation will be carried 
out In laboratory fashion under the supervision and critique of the instructor. 

Objective 8 will be conducted in an actual scHool or State Department of 
Education where the total curriculum may be viewed in operation as part of the 
educational program. A site will be selected, and in cooperation with adminis- 
trators and staff personnel, the learner' (intern) with the guidance of the 
college supervisor will carry out an actual program evaluation. The results of 
the evaluation will be forwarded to the administration and faculty of the coop- 
erating agency for their consideration. 

Evaluation by ti.e intern will differ from a real, on-going curriculum 
evaluation in that it yill be limited to: (1) time constraints, and (2) specific 
content areas. Whenever possible, the intern should be provided the opportunity 
to participate in a comprehensive process evaluation. A summary of the specific 
performance objectives for Part II are as follows: rttgg, ^ 

Performance Oblectives 
Part II 

6. Asseuble, analy25e, and interpret evaluation data and present data 
in an understandable fashion to a variety of audiences. 

7. Develop a curriculum evaluation form and evaluate and critique two 
vocational program^proposals and two vocational curriculum guides. 

8. Conduct ot assist in the development of a program or class evalua- 
. tion in an actxial school setting. 

9 . 



MODULE PREREQUISITES 

Prerequisites for the evaluation module are as follows: 

1. Background learning. 

a. Basic course In statistics 

b. Course work In measurement and/or evaluation 

c. Background course work in vocational education 

2. Experience 

a. Teaching experience In one or more vocational areas 

b. Work experience In a field outside of education desirable 

c. Previous experience In conducting program evaluations desirable 
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, PRETEST 

!• The use of specific criteria to evaluate student outcomes related to the 
students potential success is: 

A. Measurement evaluation 

B. Criterion-referenced evaluation 

C. Domain-referenced evaluation 

D. Formative evaluation 

2. The process of evaluation during the construction of a* curriculum is: 

A. Sunnative evaluation 

B. Appraisal evaluation 

C. Formative evaluation 

Norm referenced evaluation 

The process of evaluation of student outcomes as compared to national 
statistics is: 

\ Ai Norm referenced evaluation 

B. Criterion referenced evaluation 

C. Summative evaluation 

D. Goal based evaluation 

The process of evaluation of a completed curriculum is : 

A. Criterion referenced evaluation 

B. Accountability evaluation 

C. Assessment evaluation 

D. SumDiative evaluation 

's. The process of evaluation based on student outcomes as measured by the 
content and objectives/ of the course is: 

A. Grading referenced evaluation 

B. Norm referenced evaluation 
€• Domain referenced evaluation 

D. Assessment referenced evaluation 

6. The process of empirically assessing the results of an evaluation is: 

A. Measurement 
Appraisal 

C. Summation 

D. Assessment 

7. The process of appraising, assessing, and estimating the relevance and 
outcomes (student and program) of a curriculum is: 

A. Measurement ' 
Evaluation 

C. . Summation 

D. Grading 

11 - 
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8. The use of a standard to establish a student's progress or achievement Is a 

A. Model 

B. Scale 

C. Norm 

D. Relationship 

9. Th.e four types of measurement scales used In evaluation are: 

A. Nominal, correlational » Interval, normative 

B. Ordinal, nominal, Interval, ratio 

C. Correlational, experimental, normative ^ formative 

D. Normative, ordinal, formative, summatlve 

10. Two commonly used graphs to visually present evaluation data are:^ 

A. Kurtosls, frequency polygon 

B. Normal, frequency distribution 

C. Histogram, skewness 

D. Histogram, frequency polygon r 

11. The four properties used to explain a statistical distribution are: 

A. Normal, leptokurtlc, kurtosls, variation 

B. Leptokurtic, central location, skewnesa, kurtosls 

C. Central location, skewness, kurtosls, variation 

D. Cyclokurtlc, skewness, normal, leptokurtic. 

r 

12. Three types of measures of variation of a group of data are: 

A. Standard range, norm, central location 

B. Range, standard deviation, percentile 

C. Quantlle, central location, percent 
Norm, correlation, range 

13. The measurement of the degree of relationship between pairs of data Is: 

A. Correlation coefficient 

B. Central variance 

C. Quartlle deviation 

D. .Range 

14. The prediction of a score range of a student based on norm referenced 
measures Id: 

A. Student at 3rd grade - 10 month 

B. Grade reporting 

C. Grade eiqulvalent 

D. Both A and C 
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15. Given below Is a schematic 
measurement, and learning, 
the appropriate sequence. 



fpr the relationship among evaluation. 
Using the terms provided, match them In 



1— 
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1. 


A. 


Fonaatlve/summatlve evaluation 


2. 


B. 


StudenC outcomes 


3. 


C. 


Program goals 


A. 


D. 


Affective measures 


5. 


E. 


Needs assessment 


6. 


F. 


Community programs ^ 




G. 


Performance objectives 




H. 


Instructional strategies 



16. The key emphasis of the CSE model Is: 

A. Procedural consideration of decision making 

B. Self assessment 

C. Instructional objective 

D. , Rules for developing programs 

17. The CSE model Is primarily used to: 

A. Determine cause and effect relationship 

B. Evaluate the products and process of a program 

C. Measure final program outcome 

D. Measure student achievement 

18. The key emphasis of the Tab a model Is: 

A. Self assfessment 

B. Instructional goals and objectives 

C. Program relevance.- 

D. Cause and effect relations hips 

19. The Tabs Model Is primarily used tjp: 

A. Find out what works . 

B. Determine program revisions 

C. Prioritize goals and objectives 

D. Measure student achievement 



20. The key emphasis of the Tylerlan model Is: 

A. Decision making 

B. Operative analysis \ 

C. Instructional objective 

D. Cause and effect relationship 
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21. The Tylerlan model la primarily uaed to: 



A Measure st^uw-int pirograsa toward objectives > 

B. Set rules for developing new programa 

C. Determine content a\id procedures o£ inatruction 

D. Meaaure final program outcomea 

22. The key emphaaia of the CIPP model ia: 

A. Staff aelf-ev&luat/ca 

B. Provide for deciaion making 

C. Aacertain atudent progreaa 

D. Inatructional objectivea 

23. The CIPP model ia primarily uaed to: 

A. Determine content and procedurea of inatruction 

B. Facilitate continuous deciaion analyaia 

C. Find an explanations of what works 

D. Determine cause /effect relationahipa 

24. The eatabllahment of program 3tandarda (criteria) are eaaential to program 
evaluation. All of the following repreaent applicable criteria EXCEPT: 

State ind Federal atandarda 

Planning and development, organizing and adminiatration 
Staffing and operationa 
Cognitive and affective evaluation 

25. The eatabllahment of learner outcomea criteria are eaaential to program 
evaluation. All of the following rep reaent. applicable criteria EXCEPT: 

Deacription of learner needa 
Levela of accepted learner behavior 
Staff : atudent ratio 

Goala and objectivea of program » 

26. The use of observation* aa an evaluation technique ia applicable to vocational 
programa. When evaluating and/or creating an observation rating acale, which 
one of the following repreaenta criteria that muat. be uaed: 

Proficiency level and rating criteria 
Subjectivity learning and rating criteria 
Error of leniency and proficiency levela - 
Proficiency JLevela and anbjectivity 

27. When using an obs-rvatioa aheet which of the foil ing la not_ a aource of 
error: 

A. Inatructora knowledge of what to obaerve 
. B. Leniency, ego, halo effect 

C. Error of generalization, high marka for beat atudenta 

D. Int erf erea yith learning process, validity ^ 
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28. An observation sheet la a device designed to aid an evaluator to: 

A. quickly observe persons during an activity and to evaluate their 
performances 

B. observe persons and compare their performance 

C. standardize their scoring for a particular activity 

D. both A and C are correct 

29. Pretest evaluation will provide the evaluator with the following kinds 
of Information EXCEPT: 

A. Entry abilities 

B. Prerequisite knowledges 

C. Entry skills 

D. Developed knowledge 

30. Posttest evaluation will provide the evaluator with the following kinds 
of Information EXCEPT: 

A. Developed skills 

B. Increased knowledges 

C. Increased abilities 

D. Developed potential 

31. Dls tractor Is to key as: 

A. Choice : constant 

B. Option : answer 

C. Sample : population 

D. Error : replica 

32. Stem is to options as: 

A. Beginning : end 

B. Question : answer 

C. Right : wrong 

D. Condition : responses 

33. The level of difficulty for a particular question in item analysis refers t 

A. The ease with which one answers the question 
The relationship of one question to another 

C. How difficult one question is to answar by a normal distribution 
of the population 

D. How difficult one question is to answer by a sample of the population 

34. One can say that evaluation of learners knowledge about a specific subject 
can be determined if high quality: 

A. questions are developed to cover each of the performance objectives 
of that unit 

B. performance objectives have been achieved 

C. questions are developed 

D. knowledges are achieved 
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Criterion Is to standard as: 



A. has : have not 

B. what: how much 

C. behavior : who 

D. how : who 

3 6. Conditions Is to behavior as: 

A. classroom : discussion 

B. pupil : will type 

C. learn : In 5 minutes 

D. person : must sell at contracted price 

37, The term error of measurement are those errors that: 



A. Go unnoticed ^ 

B. One can predict 

C. One has no control of 

D. Occurs In connection "Ith the procedure of observing, measuring, 
counting, etc. 

38. Observable change In student beh£*vlor provides an evaluator with an 
Indication of which of the following: 

A. Intelligence of learner 

B. Effectiveness of an Instructional unit 

C. Interest exhibited by learner 

D. Teaching technique effectiveness 

39. The first step In developing a plan for an evaluation device Is to; 

A. Determine the total score or points to be given 

B. Analyze learning activities to determine what can be measured 

C. State objectives In behavioral terms 

D. Determine extent of evaluation needed 

40. A normal distribution of learners' achievement can best be established by: 

A. Giving unlimited chances to demonstrate ability 

B. Giving one chances to demonstrate ability 

C. Giving one timed chance to demonstrate ability 

D. Giving several timed chances to demonstrate ability 

41. A pretest may be^ brief and Informal If: 

A. It Is known that the learners have no prior experience or Instruction 

B. The objectives are In the psychomotor domain 
^ C. The objectives are In the affective domain 

D. Time Is unlimited 
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42. A usable pretest: 

A. Is related to the behavioral objectives 

B. Is based on prerequisite knowledge » skills^ or attitudes 

C. Is limited to paper and pencil tests of basic knowledge 

D. Is based on subjective evaluation 

43. Objectives are more useful as a guide for developing evaluation 
devices If they: 

A. Are stated In general terms 

B. Are limited to specific knowledge of facts 

C. Include descriptions of post-Instructional behavior 

D. Identify the level of performance to be achieved 

44. From a selection of foods, the learner will be able to select 3 foods 
and the amount of each of the 3 foods, which are equivalent to a 2 oz. 
serving of lean beef as a source of protein. This objective would be 
classified in which of the following psychological domaln(s)? 

A. Cognitive 

B. Affective 

C. Psychomotor 

D. Physiological 



17 



12 



PRETEST KEY 



1. B 

2. C 

3. A 

4. D 

5. C 

6. A 

7. B 

8. C 

9. B 

10. D 

11. C 

12. B 

13. A 

14. D 

15. 1 - E 

2 - C 

3 - G 

4 - H 

5 - B 

6 - A 

16. A 

17. 5 

18. D 

19. A 

20. C 

21. A 

22. B 

23. B 

24. D 

25. C 

26. A 

27. D 

28. D 

29. n 

30. D 

31. « 

32. E 

33. C 

34. A 

35. B 

36. A 

37. D 

38. B 

39. C 

40. D 

41. A 

42. B 

43. D 

44. A 
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POSTTEST 

Match the :enii8 with the most appropriate definition: 



1, 
2. 
3. 

4. 
5. 

6. 

7. 
8. 
9. 

10. 



Definitions 

^Empirically assessing the results 
on evaluation 

Evaluating a curriculum during Its 
construction 

Evaluation based on student outcomes, 
"measured by Content and objective of 
a course 

^Evaluating a curriculum after Its 

construction 
^Criteria against which student 

outcomes related to student's potential 

success 

Process of appraising, assessing, and 
^estimating the outcomes (student and 

program) of a curriculum 
^Criteria of student outcomes to 

national statistics 

Measurement of degree of relationship 



between pairs of data 
_State tod federal standards, staffing 
^and operations, planning and develop- 
ment, and administration 
Standard to establish student progress 
or achle\ srment 



Terms 

A. Formative evaluation 

B. Norm referenced 
C* Measurement 

D* Sumatlve eiNiluatlon 

E* Evc:lisatlon 

F* Correlation coefficient 

G* Do^ialn referenced 

H. Criterion ref wenced 

I. Norm 

J. Program Standard Criteria 



11. The terms ordinal, nominal. Interval, and ratio represent: 
A* Graphs 

B« Measure of variations 
C* Measurement scales 
D. Score ranges 

12. Two conmonly used graphs to visually present evaluation data are: 

A. Kurtosls, frequency polygon 

B. Normal, frequency distribution 
C* Histogram, skewness 

D* Histogram, frequency polygon 

13. The four properties used to explain a statistical distribution are: 

A. Normal^ leptokurtic, kurtosls, variation 

B. Leptokurtic, central location, skewness, kurtosls 
C* Central location, skewness, kurtosls, variation 
D* Cyclokurtic, skewness, normal, leptokurtic 



19 



Three types of measures of variation of a group of data are: 



A. Standard range, norm, central location 

B. Range, standard deviation, percentile 

C. Quant lie, central location, percent 

D. Norm, correlation, range 

The prediction of a score range of a student based on norm referenced 
measures is : 

A. Student at 3rd grade - 10 month 

B. Grade reporting 

C. Grade equivalent 

D. Both A and C 

Using the terms provided, outline a schematic of the relationship among 
evaluation, measurement, and learning. Use the letter preceding each 
term In the schematic to label the elements of your Illustration. 

A. Student outcomes 

B. Program goals 

C. Needs assessment 

D. Performance objectives 

E. Instructional strategies 

F. Formative evaluation 

G. Summatlve evaluation 



The evaluation of products and process of education programs upon which 
decisions are based are the use and emphasis of which one of the 
following evaluation models: 

A. Tylerlan 

B. CSE 

C. CIPP 

D. Popham ^ 
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18* The key eaphasls of the Taba model Is: 
A* Self assessment 

B* Instructional goals and objectives 

C* Program relevance 

D. Cause and effect relationships 

19. The Taba model is primarily used to: 

A. Find out what works 

B. Determine program revisions 

C. Prioritise goals and objectives 

D. Measure studeflt achievement 

20. Using the evaluation model to emphasize instructional objectives and 
fin' out what works is associated with which one of the follo%d.ng 
av-oors: 

A. Tyler 

B. Taba 
C* Popham 

D* Stuftelbeam 

21. The key emphasis of the CIPP model is: 

A. Staff self-evaluation 

B. Provide for decision making 

C. Ascertain student progress - 

D. Instructional objectives 

22. The CIPP model is primarily used to: 

A. Determine content and i^rocedures of instruction 

B. Facilitate continuous decision analysis 

C. Find an explanation of what works 

D. Determine cause/effect relationships 

23. The iise of, description of learner needs, acceptable levels of learner 
behavior, and goals and objectives of a program are criteria for the 
evaluation of: 

A. Program outcomes 

B. Instructor outcomes 

C. Learner outcomes 

D. Assessment outcomes 

24. The use of observation as an evaluation .technique is applicable to 
vocational programs. When evaluating and/or creating an observation 
rating scale, which one of the following represents criteria that must 
be used: 

A. Proficiency leviel and rating criteria 

B. Subjectivity learning and rating criteria 

C. Error of leniency and proficiency levels r 

D. Proficiency levels and subjectivity 

21 



16 



25. When using an observation sheet which of the following Is not a source 
of error: 

A. Instructor's knowledge of what to observe 

B. Leniency, ego, halo effect 

C. Error of generalization, high marks for best students 

D. Interferes with learning process, validity 

26. An observation sheet Is a device to aid an evalxiator to: 

A. quickly observe persons during an activity and to evaluate their 
performances 

B. observe persons and compare their performance 

C. standardize their scoring for a particular acltlvlty 

D. both A and C are correct 

27. Pretest evaluation will provide the evaluator with the following kinds 
of Information EXCEPT: 

A. Entry abilities 

B. Prerequisite knowledges 

C. Entry skills 

D. Developed krowledge 

28. One can say that evaluation of learner's knowledge about a specific 
subject can be determined If high quality: 

A. questions are developed to cover each of the performance objectives 
of that unit 

B. performance objectives have been achieved 

C. questions axe developed 

D. knowledges are achieved 

29. Option is to answer as: 

A. Choice : constant 

B. Dlstractor : key 

C. Sample : population 

D. Error : replica 

30. Condition is to responses aa: 

A. Beginning I end 

B. Question : answer 

C. Right : wrong 

D. Stem : options 

31. row difficult one question is to answer by a normal distribution of the 
population refers to: 

A. The ease with which one answers the question 

B. The relationship of one question to another 

G. The level of difficulty for a particular question in item analysis 
D. How difficult one question is to answer by a sample of the population 



22 



- , i? 

32. Posttest evaluation will provide the evaluator with the following kinds 
of Information EXCEPT: 

A. Developed skills 

B. Increased knowledges 

C. Increased abilities 

D. Developed potential ^ 

33. Criterion Is to standard as: 

has : have not 
what : how much 
behavior : who 
how : who 

34. Conditions Is to behavior as: 

A. classroom : discussion 

B. pupil : will type 

C. learn : In 5 minutes 

D. person : must sell at contracted price 

35. The term error of measurement are those errors that: 

A. Go unnoticed i 

B. One can predict 

C. One has no control of 

D. Occur In connection with the procedure of observing, measuring, 
counting, etc. 

36. Observable change In student behavior provide an evaluator with an 
indicator of which of the following: 

. A. ^ Intelligence of learner 

B. Effectiveness of an instructional unit 

C. Interest . exhibited by lear ^r 

D. Teaching technique effectiveness 

first step in developing a plan for an evaluation device is to: 

Determine the total scor^ or points to be given 
Analyze learning activities to determine' what can be measured 
State objectives in behavioral terms 
Determine extent of evaluation needed 

38. A normal distribution o/f learners' achievement can best be establisheo by: 

Giving unlimited chances to demonstrate ability 
Giving one chances to demonstrate ability 
Giving one timed chance to demonstrate ability 
Giving several timed chances to demonstrate ability 
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39. A pretest may be brief and Informal If: 

A. It Is known that the learners have no prior experience or Instruction 

B, The objectives are in the psychomotor domain 
C: The objectives are in the affective domain 
D. Time Is unlimited 

40. A usable pretest: 

A« Is related to the behavioral objectives 

B. Is based on prerequisite knowledge, skills, or attitudes 

C. Is limited to paper and pencil tests of basic knowledge 
D« Is based on subjective evaluation ^ 

41. Objectives are more useful as a guide for developing evaluation devices 
If they: 

A« Are stated In general terms 

B. Are limited to specific knowledge of facts 

C. Include descriptions of post-Instructional behavior 

D. Identify the level of performance to be achieved 

42. From a selection of foods, the learner will be abl^ to select 3 foods and 
thjB amount of each of the 3 foods, which are equivalent to a 2 oz. serving 
of lean beef as a source of protein. This objective would be classified 
In which of the following psychological domain (s)? 

A, Cognitive 

B. Affective ^ 
C« Psychomotor 

D. Physiological 




POSTTEST KEY 



1. 


C 


2. 


A 


3. 


G 


4. 


D 


5. 


H 


6. 


E 


7. 


B 


8. 


F 


9. 


J 


10. 


I 


11. 


C 


12. 


D 


13. 


C 


14. 


B 


15. 


D 


16. 





17. 
18. 
19. 
20. 
21. 
22. 
23. 
24. 
25. 
26. 
27. 
28. 
29. 
30. 
31. 
32. 
33. 
34. 
35, 
36. 
37. 
38. 
39. 
40. 
41. 
42. 



B 
D 
A 
A 
B 
B 
A 
A 
D 
D 
D 
A 
B 
D 
C 
D 
B 
A 
D 
B 
C 
C 
A 
A 
P 
A 
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GLOSSARY OF TERMS 

Crj^erlon-ref erenced Measurement - student outcomes as related In specific 
outcomes related to potential success* 

Domain'^ref erenced Measurement - student outcomes as measured by the content 
and objectives of n course* 

Evaluation - appraising, assessing, and estimating the relevance and outcomes 
of a curriculum* 

Evaluation Design - definition of conditions and procedures required to 
gather evaluation data* 

Evaluation Models - processes that establish the format procedure and methods 
for educational evaluation. 

Formative Evaluation - evaluation that occurs during the planning and develop- 
ing of a curriculum* 

Learner Criteria Outcomes - criteria against which the students* performance 
nay be evaluated to assess the degree to which the program meets the learner's 
needs. 

Mastery Learning /- paphaslzlng learning rates rather jtl^n learning levels* 

Meas urement - empirically assessing the results of an evaluation. 

Norm- referenced Measurement - students' outcomes as compared to national 
student outcome statistics* 

Fr^pgr .a Criteria Standards - criteria aga4.nst which a program may be evaluated 
to asdess the degree to which the program meets the criteria* 

Rating Scale tool used to. evaluate performance of a person or program, based 
on specific criteria and perfbrmanpe level. 

Statistics - mathematical process for using facts and figures Xn evaluation 
and measurement. 

Sunmatlve Evaluation - evaluation of a completed project or curriculum outcome* 
Test - a means of evaluating the degree to which a desired outcome was obtained* 



26 

9 



PART I • 

BASIC CONCEPTS AMD BACKGROUND 
KNOWLEDGE OF WALUATZON 



PERFORMANCE OBJECTIVE 1 

Describe and explain the terminology and concepts In educational 
'evaluation. 



Learning Activity 1-a 

Using the references listed below as a basis, write a brief 
definition for each of the terms and concepts as they relate 
to educational evaluation. 





CONCEPTS 




TERMINOLOGY 


1. 


Measurement 


1. 


Norms 


2. 


Evaluation 


2. 


Graphs, (Histogram, 








Frequency polygon) 


3. 


Formative Evaluation 










3. 


Variability (Range, 


4. 


Summatlve Evaluation 




Standard deviation. 








percentile) 


5. 


Assessment/Appraisal 








4. 


Grade . equivalent (Norm 


6. 


Mastery Learning 




referenced, Predition 








of a score range) 


7. 


Crlterlon-reiEerenced Measurement 










5. 


Central location 


8. 


Norm-*referenced Measurement 




(mean, median, mode) 


9. 


, Domain- referenced Measurement 


6. 


Relationships 








( CO -relation) 



Learning Activity 1-b 

Using the concepts and terminology in Learning Activity 1-a, 
explain their use in presenting conclusions regarding the 
evaluation of either program outcomes or learner outcomes. 
Example - Learning Activity 1-b 
Application of the concept of evaluation: 
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In presenting to an advisory committee the outcomes of a 
program, the methods for appraising, assessing, and estimating 
are presented and explained* 



Suggested References for Concepts: 

Alkln, Marvin C. "Criterion-Referenced Me asurement a nd Ot her 

Such Terms." Tr*S>E7"Mono8raph Series In Evaluation #3 * 
Center for the Study of Evaluation* Los Angeles: Univer- 
sity of California, 197A* 

Bloom, B,S*; HastlngSt J*T*; and Madans, G*H* Handbook on Forma- 
tive and Summatlve Evaluation ot Student Learning * New York: 
. McGrav Hill, Inc* , 1971o Chapters 4 and b* 

Popham, James W* Educational Evaluation * En. levood Cliffs, NJ: 

Prentice Hall Inc*, 1975* pp* 9-11, 13-15, 104-110, Chapter 7* 

Wlttrock, M*C*, and Wiley David E* The Evaluation of Instruction * 
New York: Holt, Rlnehart, and Winston, Inc*, 1970* pp* 26-32, 
259-264* 

Wiley, C*E*, and Schroeder, G*B* Roles of Participants In Educa- 
tional Accountability * Denver, Colorado: Cooperative Accounta- 
bility Project, 1974* Chapters 1 and 2* 



Suggested References for Terminology: 

Ahmann, J* Stanley* Evaluating Pupil Growth 5th ed* Rocklelgh, NJ: 
Allyn and Bacon^ 1975* Part III, Chapters 8, 9, 10* 

Cronback, Lee J* Essentials of Psychological Testing * New York: 
Harper and Row, 1970* Chapters 4, 5, 6* 

Popham, James W. Educational Evaluation * Englewood Cliffs, NJ: 
Prentice Hall* , Inc* , 1975* Chapter 6* 



V 



28 



PERPORHANCE OBJECTIVE 2 

Diagram and explain the relationships between evaluation and" other 
currlcular concepts. 

Instructional Objective 2.1 

The learner will be able to diagram and explain the relationship 
betveen evaluation, learning neasureaentt and instruction in educa- 
tional evaluation. 

Learning Activity 2.1-a 

Given the following reference develop a laodel to show the 
relationship between evaluation and instruction/learning. 

0 

Wellman, Frank E. "Systems Model for Guidance Program Develop- 
ment and Evaluation." Pupil Personnel; A Handbook for Pro- 
gram Development and Evaluation . Columbia, MO: University 
of Missouri, 1971. 
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SYSTEMS MODEL FOR GUIDANCE PROGRAM DEVELOPMENT AND EVALUATION 



Differentiated with respect to In- 
dividual Differences and Situational 
Variables 



Formulation of What Needs 
to be Done to Achieve Ob- 
jectives 



Developmental 
and 
Social 



NEEDS 




GOALS 


-> 



DEVELOPMENTAL 




BEHAVIORAL 


► 


PROCESS 


► 


PROCESS 


OBJECTIVES 


-> 


OBJECTIVES 




OBJECTIVES 




STRATEGIES 



0 
•H 

U 
(0 

H 

I 



0) 



0 
•H 

U 

3 

H 

I 



C 
0 
O 
0) 

CO 



0 

1-1 

H 

I 



I 

ill 

M 
•H 

(14 



OUTCOMES 
(Evaluative Criteria) 


^General- 


# Conceptual 


• Perceptual 


ization 








termedi- 


• imnedia&e 


^Ix)ng 


*ate 


• Behavioral 


Range 






• Decisions 




Global 




0 Criterion 


C develop- 


• Plans 


Referenced 


ment 








• Adjustment 


^Norm 


perform- 




Referenced 


ance 








Performance 




adjust- 


• (Broad 




ment 


Based) 





PROCESS 
ACTIVATION 



J 



Wellman] Frank E. "Systems Model for Guidance Program Development and Evaluation." Pupil Personnel : 
A tjandbook for Program Development and Evaluation . Columbia, HO: University of Missouri, 1971. 
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28. An ob8e^^^|^j^og^^|J^^^i|„a^^e^5g,dgg^^ft^t8 8t^Maaer«l»^'=°'^ to: 

A. quickly observe persons during an activity and to evaluate their 

The systl&I^SBSSi^for guidance program^ development and evaluation provides a 

B, Observe persons and compSre' tnelr perrbrmance 

r£-fof |R^^J^EInitlSi£xIH5i"8f^efie^g8iSii&a^|fi|fU^i^^ of (l) general 

n. both A and C are correct i 

~d29?^^^?l8e??P|^a?^!5S^^5!ir^r8^?!^ l5ilil^o?^^ffg*tR|Jf§f!j5fnf^lnds 
'strategies iesJIne&^^o acfel2?6 the pupil objectives. The flow moves from the 

general |o t|g^s|e|||Jf^^^h5^ge^ of objective^, with the more specific 

represen^lng|^|||^ep||i^8^o|^^|ectlve8 and outcomes where the program should make 

willing l3£u£S^f4%]i^SSIIity and to be held accountable. The following expla- 
nations tSl^villoW^oiiSnents of the model are directed toward (1) working 

B . incr^ased^ knowledges 

deflnltlgns,JaSS^J?|^rl^|U^ in the systematic flow from need to outcome. 

1. Needs. Needs are defined here to Include those long range developmental 
31. D isc r a ctor is to Key as: 

needs of individuals and the general social needs of the society that 
A. Choice ; constant 

1ustify^li4?SxistM<5l^pf a guidance program. They provide the underlying 

C. Sample : population 

ratSbna5£^?SrVtR|PHS|'^™ and the basis for developing goals and objectives. 

^^SucR^neeil sftoui&^Se^rllAted to the guidance program but by their broad 

clonal na?u9S^S&y'ln^^lve maky other programs, people, knd circumstances. 
^ B, Question i answer 

For^example^ k IJe^Slopmental need of individuals might to become a 

D. Conaition : responses 

33?«'^fl'flvl¥^oP'^S^?fJlSftrFof'a pi^!lSfflf"f5elff5S«fn«5?P^^aSI??lft^'lfer8 to 

DHol%f?lflt°iXiS^^^^^ ISgJfe''o°rMI p^o^^uten 

theSfitilal*iS?aMiiL^ife^t*'MH&^iy related to the guidance 'program 
lnclude^JI|||f5e||loJ§S^mio&?eSr^^^ 

the|tl^J^I|jy|n^^|J^JX|^|teo%S«at^t^^^^^ The theory or theories 

tha'^-unfe^lMsiAy^lLScli^l^ltince program should be Identified and used to 
build the rationale for the development of pupil goals and objectives, Indud- 



lug th% part that guldanca actlvltlaa may play In tha total educational, 
vocational, or 9ccial davalopmant of Indlvlduala. Tha baalc aodal naada 
ara laaa directly related to pup 11-* centered guidance object Ivea, but ahould 
have a direct bearing upon the program rationale* For example, individual 
economic a^lf-auff Iclency may repreaent a general aoclal need related to 
contrlbutlona of the guidance program, but again the broad and remote 
nature of needs auggeeta that there will be many factora, other than 
guidance, influencing the outcome. Specification of needa la desirable 
to build a foundation from vhlch goala and objectives can be atated In 
more meaningful and manageable terms. 

2* Goals . Goals are the reasonably long range operational delineations 
of needs* They are more specific than needs, but still too far removed and 
too confounded by other Influences to serve as workable objectives for the 
guidance program* Goals take on relevance for Indlvlduala, In that they 
will vary among Individuals and from one situation to another* The general 
need for all Individuals^^ become vocationally competent Is thus defined 
In terms of goals that specify that particular Individuals complete a 
training program that will qualify them to enter a specific occupational 

area that Is consistent with thefr abilities and Interests* Goals represent 

-» 

rather large global areas of achievement or development that lead to 

meeting specified needs, and provide the point of departure for the 

formulation of developmental and behavioral oSjectlves^ 

3* Developmental Objectives * The developmental objectives relate to the 

guldance-*related developmental steps that are logically, and usually 

sequentially, related to the stated goals* For example, before an Individ** 

ual can achieve the occupational competence goal stated above, there 

are a number of vocational and educational decisions that need to be made* 

Each auch decision can become a developmental objective for the guidance 
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program. Likewise, a aeries of plana nay need Co be forvulated and these 
can becone objectives* Certain situatdtoilal adjustments may also relate 
to the goal, as may intermediate range performance, such as coaoipletion of 
prerequisite courses and curricula. Thus, many developmental objectives 
may be formulated to operationally define the hypothesized guidance con-* 
tributions to a single goal. Again, it is easy to see that the achievement 
of Mny developmental objectives may. be influenced by other factors, and 
the related guidance activity may be only one small part of a rather complex 
process. There are, however, many developmental objectives for which 
guidance personnel should be willing to assume^^major responsibility. 
Developmental objectives have particular significance in differentiating 
desired outcomes for different age-and grade groups. For example, the 
objective to develop an appreciation for work in everyone's life may be 
quite relevant for a primary grade group, while the junior high school 
group may need to develop an appreciation for work as a very personal 
part of their lives. Each developmental objective should be stated and 
related to program so that the part that guidance may contribute is clearly 
understood. This can be done through the formulation of behavioral objec- 
tives. 

4. Behavioral Objectives . Behavioral objectives should specify in concrete 
terms the knowledge, skill, or performance that is expected. These objectives 
should be related directly to a specific guidance activity and at least 
logically to a developmental objective. They are usually short range in 
terms of time and should always be related to specific criteria for 
determining whether the objective has been achieved and often the degree 
of achievement. The developmental objective to make a decision regarding 
post high school education may provide the basis for specifying relevant 
behavioral objectives spelling out the kinds and extent of awareness of 

34 
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self and environment needed to make the decision. The developmental 
objective requires that Ixidlvldfials conceptualize self In the educational- 
vocational situation. This conceptualization requires that they be aware 
of their abilities. Interests, resources, etc., and of the environmental 
opportunities, requirements , etc. , and that they be able to demonstrate such 
awareness by showing that they have specific knowledge of these relevant 
variables. Behavioral objectives are crucial to adequate program develop-- 
ment, provide the most objective basis for evaluation, and make develop- 
mental objectives and goals meaningful by defining sequential developmental 
relationships in terms of behavior that can be described from observation, 
objective measurement, and self-reports. For example, comprehensive 
career development theory can provide .the basis for determining behavlora^ 
objv r.tives appropriate to the maturity levctl of the pupils and related to 
the sequential developmental objectives that lead to the longer range 
vocational goal. (See examples of vocational goals from the chart on the 
following pfage. ) 
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GOALS 

T 

Adapted to Indlviduili and Situation 
(Ganeralisatlon level behavlon) 



Fomilated froa idtntiflad dmlopMDtal 
and aoclal ottds - appropriate to the 
itudent population and the fuoetiooal 
iituation. 



i 

Educational 

Ezaaplei of goali: 

1. Regularity of school attend- 
ance or reduction of abiencee. 

2. Individual, and group contri- 
butiona to the school cob- 
■unity - constructive partici- 
pation - reduction of inter- 
ruptions to order » etc. 

3. Value education and develop 
aapirationa consistent with 
abilities and interests. 

4. Successful coapletion of 
successive educational pro- 
graas or reduction of pre- 
Mture dropouts. 

5. Pursue higher levels of 
education consistent with 
individual level of gen- 
eral and specific abilities 
and interests. 

6. Seek out and participate 
in nev educational « 
opportunities. 

7. Value and suppcrt educa- 
tion .throughout life. 



^ — ^ 

Vocational 

Exaaples of goals: 

1. CoMdtatnt Co a chosen voce- 
tiob. 

2. Participation in and com- 
pletion of education pro- 
graa necessary for entrance 
in chosen vocation. 

3. Individual derive personal 
satisfaction froa the pur- 
suit of a vocation. 

4. Vocational Access ** e^>loy- 
aent - satisfaction - pro- 
gress. ^' 

5. Denrelopaent and utilization 
of individual abilities and 
interests through vocational 
pursuits. 

6. Reduction of social probleas 
related to vocational adjust- 
•ent: 

(a) uneaployaent 

(b) undereaployaent 

(c) poverty 

(d) crlM 



Social 

Bza^)lea of goals: 

1. Develop and use effectively 
behaviors consietsnt with 
sociel deiinds - faaily - 
peers - school * coHnmlty. 

2. Develop and display appro- 
priate social roles in 
tens of setting and 
grpup. 

^. Value social interaction 
and socially acceptable 
experiences. 

4. Contribute to the social 
grovth and effectiveness 
of others. 

5. Develop openness to the 
ideas, values, and experiences* 
of others. 

Continue to grov through 
social experiencu. 
7. Becoae a socially respon- 
sible and contributing 
citixen. 



Vellaan^ Frank E. ^ "Systou Hodel for Guidance Prograx Developeent and Evaluation." Pupil Perso nnel; A Hand- 
book for .Program Developnent and Evaluation . Colmbia, MO: University of Missouri, 1971. 
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Specifying Workable Objectives and Achievable Outcomes ^ 



The specification of meaningful objectives and the us^ of appropriate criteria 
to estimate the achievement of objectives are essential for. adequate evaluation. 
The difficulties encountered In accomplishing these tasks have, no doubt, dls* 
couraged many guidance vorkers from undertaking comprehi^nslve outcome studies* 
The basic requirements for stating vocational guidance goals, objectives, and 
outcomes follow the major parts of the systems model with' the added criterion of 
feasibility imposed at each step in the process. 

1. Objectives should be oriented to' identified student needs (educational, 
vocational, and social.) 

2. Objectives should be consistent with societal values and' professional 
philosophy* 

3. Objectives should be stated so they can be traqslated into expected 
^ behavioral outcomes (relevant to the primary purposes of vocational 

guidance* ) 

4* Behavioral outcomes should be defined operationally so that they can be 
quantified in terms of knowledge, skills, performance, and attitudes* 

5* The data needed and methods for measuring and reportjlng behavioral ^ 
manifestations should be specified for each objective and outcome* 

6. Objectives and outcomes should meet the test of relevancy for the 
student sample (such as grade level, sex, etc.) and the operational 
situation (such as type of school, demographic characteristics, etc.) 

A framework for the specification of guidance objectives is part of the pro-* 
posed National Study of Guidance* This taxonomy may serve as a guide for the 
specif Icf. Aon of vocational guidance objectives and outcomes that meet the above 

1 

Wellman, Frank E* "Systems Model for Guidance Program Development and 
Evaluation." (Excerpts from a paper presented for discussion at 64th An^ 
American Vocational Association Convention, New Orleans, Louisiana, De ar 
7, 1970. ) 
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r«qulr«Mntt and those of thm wyBtnm nodel. Tha follovliig outline Bhaum the 
nijor categories Of giildance objectives Included In this taxonongr. 
1.0 Perceptualltatlon Objectives . The development of awareness and differenti- 
ations of relevant envlronaental and self variables. 

1.1 Envlronaental Orientation . Knowledge and understanding of educational, 
vocational, and social opportunities, requirements, and expectatlona. ' 

1.2 Self*-Orlentatlon . Knowledge and understanding of abilities, llnltatlons 
Identities', feellugs, and notlvatlons relevant to educational, vocational 
and social development.' ^ 

2.0 Conceptualization Objectives . The process of analyzing relationships, making 
predictions, evaluating consequences, and taking actions relevant to educa* 
tlonal, vocational, and social goals. 

2.1 Directional Tendencies . Formulation of decisions and plans, fnd the 
development of Interests and value attachments which result In Increasing 
stability and consistency In movement toward educational, vocational, 
and social goals. 

2.2 Adaptive and Adjustlve Behavior . Development of coping behavior to meet 
' educational, vocational, and social requirements and expectations. 

3.0 Generalization Objectives . The development of a behavior pattern typified 
by consistency, commitment, effectiveness, and autonomy. 

3.1 Accommodation . The psycho-social ability to cope %d.th cultural and 
environmental demands. 

3.2 Satisfacti on. The Internal self Interpretations of environmental 
transactions. . 

3.3 Mastery . The congruency of expected or predicted achievement with 
external criteria of achievement. 

The following example Illustrates how the taxonomy can be used for the 
bpeclflcatlon of objectives and outcomes.. Assuming a group, of tenth grade 
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boys In a comprehensive high school and using objective 2.2-1 from the voca- 
tional domain of the taxonomy^ the resultant organlxat|Lon for evaluation 
might be: . 

Objective: For the student to Identify occupational alternatives that are 
consistent with his abilities and claimed Interests. ^ 
Ten week occupational orientation supplemented with three 
Individual counseling conferences with each boy. 
Each boy will be able to list at least one occupation consistent 
with his abilities and claimed Interests » and 75Z of the boys 
will be able to list five or more such occupations. 



Process: 



Expected 
Outcome: 



References: 

Bloom, B.S.; Hastings, J.T.; and Madans, G.H. Handbook on Forma- 
tlve and Summatlve Evaluation of Student Learning . New York: 
McGraw Hill Inc., 1971. Chapters 4 and 6, 

Moss, Jeroine Jr., and Strpmsdorfer, Ernst W. "Evaluating Voca- 
tional and Technical Education Programs," Vocational Educa- 
tion: Today and Tomorrow , Editors Gerald Somers and Kenneth 
Little. Center for Studies In Vocational Education, Univer- 
sity of Wisconsin, 1971. Chapter 9. 

Payne, David A. The Assessment of Learning: Cognitive and Affec- 
tive . Lexington, MA: D.C. Heath, 1974. Chapter 1. 

Popham, James W. Educational Evaluation . Englewood Cliffs, NJ: 
Prentice Hall Inc., 1975. Chapter 4. 

Tyler, Ralph; Gagne, Robert; and Scrlven, Michael. "Perspectlons 
on Curriculum Evaluation," AERA Monograph Series on Curricula 
Evaluation #1 . Chicago, XL: Rand McNally, 1967. 

Wandt, Edwin and Brown G.W. Essentials of Educational Evaluation . 
New York: Henry Holt and Co,, 1967. Chapter 1. 

Wellman, Frank E. "Systems Model for Guidance Program Development 
and Evaluation," Pupil Personnel; A Handbook for Pro gram 
Development and Evaluation , Columbia, M): Unlver 'ty of 
Missouri, 1971, 

Wlttrock, M.C., and Wiley, David E. The Evaluation of Instruction . 
New York: Holt, Rlnehart, and Winston, Inc., 1970. Chapter 1. 
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PER FORMANCE OBJECTIVE 3 ' 

Explain the key salient features and usage of evaluatjLon nodels having 
relevance for vocational education. 



Instructional Objective 3*1 

The learner will be able to describe the usage and emphasis of four 
.evaluation models* 

Learning Activity 3*l"a 

Consult the list of references and complete the *'Grld on 
Prototypes of Curriculum Evaluation." 

§ 

References: 

Bloom, B.S.; Hastings, J.T.; and Nadans, G.H. Handbook on Forma- 
tlve and Summatlve Evaluation of Stucjent Learning . New York: 
McGraw Hill Inc. , 1971. Chapters 2, 4, 6. 

Popham, James W. Educational Evaluation . Englewood Cliffs, NJ: 
Prentice Hall Inc., 1975. Chapter 2. 

Taba, Hilda. Teaching Strategies and Cognitive Functioning in 
Elementary School Children . Cooperative Research Project 
No. 2404. San Francisco, CA: San Francisco State College, 
1966. / 

/ ** 
Wellman, Frank E. "Systems Model for Guidance Program Develop- 
ment and Evaluation," Pupil Personnel 1 A Handbook for Program 
Development and Evaluation . Colusibla, M): University of 
Missouri, 1971. 

/ 

/ 

/ 

/ ■ 
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. GRID ^ w 

PROTOnPES F0^CURRICULUM EVALUATION 



V / OUTSIDE / 
KEY KEY^-/ EXPERTS TEACHING STAFF 

MODEL EMPHASIS KEY USE ACTIVITIES NEEDED INVOLVEMENT RISKS APPLICATION 



1. School 

accreditation 


staff 

self-study 


review and 
content 

of Instruc- 
tion 


discussion & 
professional 
jUQgmenco 


none - 
unless / 
otner 
profes- 
sionals 


high Involvement 
essential, evalu- 
ation based on 
staff lnq>act 


subj active, 
extensive, 
time In- 
volvement 
of staff. 


involves staff lUi 
leadership and 
program/Instructional 
evaluation 


2. Cause and 
, Effect 
H. Taba 
















3. CSE 

Al Klnetal 
















4. CIPP 

Don Stufflebeam 
















5. Goal Attain- 
ment 
R. Tyler 
















OPTIONAL 

6. Formative - 
Sumnatlve 
Bloom, et. al 
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PERFORMANCE OBJECTIVE A 



• ••• 



Select and/or develop criteria that may be applied in evaluation of 
vocational education curricula* 



Learning Activity 4-a 

Using the examples^provided by the instructor, distinguish 
between "program standards criteria" and "learner outcome 
criteria;" state how they interrelate, and their relevance 



*The examples, which follow, are: 

1. Off ensiveness' Criteria 

2. A Sunraary of Educational Evaluation Guidelines 

3. Elements of a Quality Career Guidance , Counseling 
and Placement Program 

4. Career Development Program: An Example 



for evaluating vocational educational curricula. 
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Offenslveness Criteria 



The following classifications of subject matter indicate the types of material 
considered offensive: 

1. Invasion of privacy . Questions in any way connected with family financing 

are offensive. Included would be questions dealing with how much an indi/id- 
ual earned or received as an allowance, how income is budgeted, what 
contributions are made, and what taxes are paid. 

Exercises dealing with parents' relationships with their children. Included 
vrould be questions such as, "When was the last time you had to scold or 
punish one of your children for not minding you?" 

Efforts to observe actual behavior with respect to health and safety rules. 
For instance, the inspection of homes and cars to see if they are in a 
safe condition, or the attempt to observe children in washrooms to see 
if they followed health rules pertaining to the washing of hands would 
be considered offensive. 

Asking teachers, counselors, or employers to rate an individual's perfor- 
mance as compared to the performance of others was considered an invasion 
of privacy. 

2. Ethnic minority groups . Material which might be Interpreted as demeaning 
the blacks (old Negro dialect) or some other minority group was considered 
offensive. 

References to specific minority groups should be eliminated whenever 
possible. There should not be an over-weighting of exercises dealing 
with race or blacks. More exercises about other ethnic and minority groups 
should be included. 

I 

Criteria utilized by lay panels who evaluated the possible effectiveness 
of items included in exercise in the NAEP program: The National Assessment 
Approach to Exercise Development * pp. 42-46. 
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3. Sex , Literary passages with sexual references or thenes, such as Oedipus 
Rex, should be very carefiaiy chosen as they have a great potential for 
of f enslveness • 

Questions dealing with birth control, unless very carefully stated, 
should be avoided. 

4. Religion > The area of religion was considered so sensitive that It could 
not be handled without emotional reaction. 

Reference to the "God Is Dead" movenient was considered too controversial 
to use. 

However, religious music could be used on tapes as long as one did not 
have to Identify It or relate It to religion. 
Religious subjects could te used on art plates. 

5. Human rights . Exercises dealing with human rights must be worded carefully 
to reduce implications which are offensive. For Instance, the statement, 
"The nuid>er of murders Increased In a neighborhood where blacks and whites 
lived together," can be Interpreted to imply that there Is more violence 
wh^re blacks are. Another exercise implied that only the "poor" steal. 

The overabundance of exercises dealing with a person's rights was considered 
offensive unless more exercises were added dealing with their reBponslblll- 
ties in a free society. 

6. Emotion- arousing terms . Any reference to sex, unwed mothers, divorce, 
whiskey, the FBI, the President, Communism and specific organizations, 
such as the Ku Klux Klan and labor unions, might make an exercise offensive 
unless extreme care was used in the wording. The terms "protest meetings" 
and "demonstrations" apparently evoke emotional reactions. Preferable 
wording might be "public meetings on behalf of" or "in protest of." 

7. Violence or cruelty . Exercises dwelling on violence or cruelty were 
considered offensive. 
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8. Words In poor taste > Passages with certaih words or phrases were considered 
Inappropriate for iise In the assessment (6«g«, a poem that Includes the 
line, "sportive ladles leave their doors ajar,** or references to dope, 
divorce and drunkenness for ages 9 and 13). 

9. Censorship * Affective exercises In the area of censorship which Indicate 
a restriction of a family member (e.g., **my brother**) should be avoided. 
Questions such as **I would not allow my child to see a movie In which 
there Is an adulterous love affair** are personal questions about one's 
behavior and are emotion-arousing, so are felt to' be an Invasion of 
privacy. 

Statements about censorship In general are less offensive. **I think 
movies with adulterous love affairs should be banned** Is an expression 
of an objective opinion. Tt does not threaten one's privacy, and hence 
Is much less offensive. 

10. Uncomplimentary conments about specific groups or Individuals . Exercises 
which show national heroes in an uncomplimentary fashion, though factxxally 
accurate, are offensive. 

Exercises which might be Interpreted as putting the police or. other 
authorities in an unfavorable light are offensive. 

Negative statements about any group (e.g., **A11 politicians are corrupt.**) 
or attitude statements which infer that a group of people are peculiar 
(e.g. , scientists) are offensive. 

Exercises about highly controversial figures, such as the late Senator 
Joseph McCarthy, should be neither too critical nor too favorable. 

11. Inferiority of other nations. Any exercise Implying the inferiority of 
other nations or exercises which imply the superiority of Americans to 
people in less well developed countries are offensive. 

12. Questioning one child about another's behavior . The suitability of obtaining 

-) * ■ 
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Infomatlon froia one youngster about another was strongly questioned, as It 
would seen to encourage "tattling" and would probably not obtain any 
Information which couldn't be obtained In a more reliable way. 

13. Darwinian theory . Questions which require the Interpretation of Darwinian 
theory were considered objectionable. However, there was no .objection 

to factual questions regarding the nature of Darwin's theory. 

14. Civil War . In exercises about the Civil War, care must be taken to elimi- 
nate terms such as "hatred," "forced upon the South," or Implications that 
the Horth was better than the South. 

An example of an exercise which was dropped because the lay reviewers felt that 
It Implied the Inferiority of another nation was the following literature 
exercise. ^ 
Dltvxtlons: 

Read the paragraph below carefully and then circle the letter by the 

y 

chararter or tale that you thlnic the story Is based on. 

When the explorers arrived at the head of the river, they were 
surprised. They'd expected the natives to be dirty thieves, cannibals 
even. But they were treated with greater courtesy than they wuld 
have been In the United States. These people were Ignorant, yes, but 
clean, generous, and highly moral. 

(a) The New Breed 

(b) The Noble Savage 

(c) The White Man's Burden 

(d) The Old Adam 
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A Summary of Educational Evaluation Guidelines 



excerpt from 



Popham, .W. James.. An Evaluation Guidebook * Los Angeles, CA: The Instructional 
Objectives Exchange, 1972. V 



1. The educational evaluator should encourage the use of Instructional 

objectives which provide explicit descriptions of the post-Instructional 
behavior of learners. 

2« While recognizing that nonmeasurable goals will be of limited use for 
their purposes, educational evaluators must be aware that Instructors 
may wish to devote a reasonable proportion of their efforts to the 
pursuit of Important but currently unassessable objectives. 

3. The educational evaluator must Identify criteria of adequacy when using 
Instructional objectives which require constructed responses from 
learners. 

4. The educational evaluator should foster the use of measurable objectives 
which possess content generality rather than test Item equivalence. 

5. Prior to the Introduction of the Instructional treatment, educational 
evaluators should strive to establish minimal proficiency levels for 
Instructional objectives. 

6. The educational evaluator will often find the Taxonomies of Educational 
Objectives useful both In describing Instructional objectives under 
consideration and In generating new objectives* 

7. The educational evaluator should consider the possibility of selecting 
measurable objectives from extant collections of such objectives. 

8. The educational. evaluator should avoid the use of norm-referenced 
measures, preferring Instead criterion-referenced measuring devices* 

9. The educational evaluator should use domain-referenced achievement 
testing procedures when called on to generate test Items for objectives. 
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or to determine the content validity of ulready developed test items* 

10. The educational evaluator should encourage the use of multiple criterion 
measures by generating Alternative schemes for assessing learner 
behaviors • 

11. When the use of customary measuring tactics may produce reactive effects, 
the educational evaluator should employ unobtrusive measures. 

12. The educational evaluator should be particularly attentive to unan- 
ticipated outcomes which result from an instructional treatment » and 
should use such outcomes In Judging the merits of the treatment. 

13. The educational evaluator should clarify value preferences of various 
groups regarding desired educational goals by having sets of precisely 
stated objectives rated by the individuals involved, then translating 
these ratings into composite indicators of each objective's worth. 

14. The educational evaluator can compare preference and performance data 
by contrasting preference ratings of objectives with measures designed 
to assess the degree to which learners have achieved those objectives. 

15. Whenever large scale measurement is required, the educational evaluator 
should consider the economic advantage of employing item sampling and 
person sampling procedures. 

16.. In conducting formative evaluations, the educational evaluator will find 
it useful to (1) employ small samples of learners, (2) secure performance 
data regarding terminal and en route objectives, (3) use designs such 
as the one shot case study or the one ^oup pretest-post-test design. 

17, In conducting summative evaluations, the educational evaluator should 
prefer the use of designs involving randomized control groups, that 
is, the pre test-post test '^ontrol group design or the posttest only 
control group design. When randomization is. impossible^ the evaluator can 
profitably employ the nonequivalent control group design or the inter- 
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rupted time series design. 

18. The educational evaluator should analyse data according to the smallest 
Independent units available, frequently lading to the Use of classroom 
or larger units rather than individual pupil units. 

19. The educational evaluator should, in general, prefer descriptive 
statistics and estimation procedures instead of statistical hypothesis 
testing ptocedures. 

20. The educational evaluator should present decision-wkers with a wlde^ 
\^ range of pertinent information so that choices among alternatives can 

be made in a cost/effectiveness context. 
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Elements of a Quality Career Guidance, 
Counseling and Placement Program* 
ELEMENT #1 - Program Planning and P^velopment 

1.1 Philosophy, assumptJo.rss, and conceptual foundations 

1.11 Does the program have i stated philosophy? 

1.12 Was the philosophy zl statement developed prior to 1968? 

1.13 Are program assumptions tied to a theoretical foundation? 

1«1A Is the program built on performance objectives in terms 
of both process and outcome? 

1.2 Initial needs assessment 

1.21 Can the program planning team be Identified? 

1.22 Were community attitudes and expectations toward career 
guidance » counseling, and placement programs assessed?' 

1.23 Were educators* attitudes and expectations toward 
career guidance, counseling, and placement programs 

messed? 

ELEME^* - Organization and Administration 

2.1 Organizational^ structure or pattern 

2.11 Can the type of organizational model Implemented be 
Identified? 

2.12 Can lines of responsibility, authority, and accountability 
be Identified? 

2.13 Can lines of communication within the operational ataff 
be described? 

2.2 Management style 

2.21 Can the management style be Identified? 

2.3 Funding an4 allocation patterns 

2.31 Can the funding needs be identified? 

i 

2.32 Can responsibility fot resource allocations be identified? 

2.33 Can tasks, assignments be identified? 



2.4 Facilities* equipment, and materials 

2*41 Can facilities, equipment, and materials be described? 

2.42 Can the age and condition of the above be described?/^ 

2*43 Can the type and number of clerical support staff ^e 
described? 

ELEMENT #3 - Staffing 

3.1 Pre-entr^ training and experience 

3.11 Do prerequisites exist for all levels of staff working in 
the career guidance, counseling, and placement program, 
including professional, peer, clerical, and volunteer? 

3*12 Do the recruitment procedures exist for securing trained 
staff at all levels? 

3*13 Does the recruitment procedure reflect program needs? 

ELEMENT #4 - Program Operations 

4*1 Scope, sequence, Iki^d distribution 

4.11 Is the^ program implemented at all grade levels? 

4*12 Does the program effectively impact all students, inclujfling 
special needs of special or unique student populations^ 

4.13 Are career development activities arranged in a develop- * 
mental sequence? 

4.2 Patterns of program articulation, communications, 
and interaction within the school system 

4.21 Is there articulation and interaction with the total 
curriculum? 

4*22 Is the program interpreted to the total school community? 

4.23 Is the program articulated ^th supporting program, e*g«, 
school psychology, special education? 

4*24 Is the program articulated across the grade levels? 

4.25 Are inter-program referrals within the school system 
facilitated? 

4.3 Articulation, coordination, and liaison between the 
school and coimnunity systems . 

4.31 Is the program articulated with community agencies? 



4*32 l8 use made of outside referral agencies? 
4*33 Is feedback elicited from connunlty agencies and utilised? 
4*4 Strategies for program change and revision 

4.41 Are methods for l]Q>lementlng needed program revision Iden- 
tified? 

4.42 Is the basis for program revision continuous, annual, or 
otherwise Identified? 

4.43 Are relrlslona future oriented. I.e., anticipatory rather 
than reactive? 

ELEMENT #5 - Evaluation 

5.1 Continuing needs assessment and evaluation 

5.11 Are needs assessed on a continuing basis? t 

5«12 Is data maintained with which to assess needs? 

5.13 Can sources of feedback information and data used to 
evaluate local programs be Identified? 

5.14 Can type and sophistication of program evaluation 
' and research models be Identified? 

5.15 Is there articulation with planning and development 
phases of the program? 
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CAREER DEVELOPMENT PROGRAM: ^AN EXAMPLE 

1.0 GOAL: SELF (AWARENESS/ IDENTITY) . To understand self In relation to the 
nature of occupations In order to plan a career. The major focua empha- 
sizes both an underst^dlng and acceptance Lf personal characteristics 
through a developmental process toward the utilization of self-knowledge 
In formulation of career planning (Wellman, 1971). 

LEVEL K-3 

DEVELOPMENTAL OBJECTIVE: 

1.1 To develop an awareness of personal characteristics. 
BEHAVIORAL OBJECTIVES: 

1.11 To describe orally one's personal characteristics. 

1.12 To describe orally how one differs from and resembles others. 

1.13 To describe orally the behavior that accompanies basic fee^llngs. 

1.14 To describe orally one's current Interests. 
PROCESS OBJECTIVES AND STRATEGIES: 

1.11 Use Illustrated films, books and graphs to provide basis for Identifying 
parts of the body and teaching vocabulary terms (Gysbers and Moore, 1971)^. 

1.12 Ask Individual to look In full length mirror and describe self (Gysbers 
and Moore, 1971)., 

1. 13 Ask Individual to Identify vocabulary words associated with pictures 
depicting various emotions (Gysbers and Moore, 1971). 

1.14 Provide an opportunity for an Individual to show and tell what he/she 
liked and/or disliked (Gysbers and Moore, 1971). 

1.15 Ask Individual to look at a photo of self and reproduce the Image (Gysbers 
and Moore, 1971). 

1.16 Ask Individual to trace body outline on large poster paper. 

1.17 Ask Individual to measure and chart weight and height. 
OUTCOMES: • , 

1.11 Ability to describe appearance from photo of self. 

.■55/ 
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1.12 Ability to name ways In which people differ. 

1.13 Ability to describe five vocabulary words related to feelings. 

1.14 Ability to rank order a list of ten activities according to personal 
preference (Gysbers and Moore, 1971). 

EVALUATION: 

1.11 Establish pre- and post-test differences. 

1.12 Establish grade level criterion standards. 

1.13 * 

1.14 * 
LEVEL 4-6 

DEVELOPMENTAL OBJECTIVES: 

1.2 To develop ability to assess own characteristics. 
BEHAVIORAL OBJECTIVES: 

1.21 To verbally differentiate between strengths and weaknesses In academic 
disciplines, measured by stands^'dlzed tests (Gysbers and Moore, 1971). 

1.22 To demonstrate understanding of own physical abilities and limitations 
by verbally differentiating between activities which Individual can and 
cannot perform (Gysbers and ttoore, 1971). 

1.23 To describe own self In terms of Interests, abilities, values, etc., 
as they relate td work (Wellman, 1971). 

1.24 To describe own characteristics that others like. 
PROCESS OBJECTIVES AND STRATEGIES: 

1.21 Have Individuals meet with their teacher to discuss their achievements, 

abilities, and limitations. Large and small group discussion of same. 
1»22 Administer teacher-made test vhlch measures skill In several areas. 
1.23 Give individual the opportunity to work with the Hall Occupational 

\ * The blanks provide space to begin developing a new model for career development. 
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Orientation Inventory. 
1.2A Have Individuals discuss with peers and teachers what characteristics 

others like In them. 
OUTCOMES : 

1.21 Ability to rank order a list of subject disciplines according to relative 
strengths: (Gysbers and Moore, 1971). 

1.22 Ability to designate from a list of physical exercises those which they can 
and cannot perform, with one example of each. 

1.23 Ability to list general descriptive characteristics, work descriptive 
characteristics related to various occupations, and characteristics 
that describe self. 

1.24 * 
EVALUATION: 

1.21 Determine percentage of Individuals (75% required for objective achieve- 
ment) meeting criterion standard for grade group (Wellman, 1971). 

1.22 Compare Individual teacher observations of likable characteristics. 
LEVEL 7-9 

DEVELOPMENTAL OBJECTIVE : 

1.3 To assume responsblllty for continuous self appraisal (Gysbers and Moore, 

1971). 
BEHAVIORAL OBJECTIVES: 

1.31 To demonstrate knowledge about self and the dynamic nature of personal 
characteristics by Identifying recent changes In interest. 

1.32 To demonstrate ability to gather self-Information from both Internal/ 
external sources by writing a self-appralsal (Gysbers and !loore, 1971). 

1.33 To assume responsibility to m&ke a positive effort toward Improving a 
personal weakness Identified in a process of self appraisal (Gysbers 
and Moore, 1971). 

1.34 To recognize the relationships between personal characteristics and 
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broad occupational clusters (Wellinan» 1971). 
PROCESS OBJECTIVES AND STRATEGIES: 

1.31 Have individual exhibit knowledge of self in a written report-style 
activity. 

1.32 Use interview techniques to gain information. 

1.33 Provide opportunity for small group discussion. 

1.34 Provide opportunity to experience simulation games. 

1.35 Provide opportunity for role-playing and field trips. 
OUTCOMES : 

1.31 Self-estimate of the desired level of occupation is more consistent 
with the accessible level of occupation. 

1.32 Aspiratioxial and achievement self-estimations have few inconsistencies 
(Wellman, 1971). 

1.33 Aspirational and achievement self-estimation is more congruent with 
teacher/counselor estimates of realistic aspirations for the individual 
(Wellman, 1971). 

1.34 * . 
EVALUATION: 

1.31 Compare pre/po©t differences between aspirational and achievement 
self-estimates (Wellman, 1971). 

1.32 Have individual list five personal characteristics related to occupa- 
tional clusters. \ * 

1.33 * 

LEVEL 10-12 
DEVELOPMENTAL OBJECTIVE: 

1.4 To formulate tentative career plans Consistent with knowledge of self. 
BEHAVIORAL OBJECTIVES: 

1.41 To make a tentative career choice tjhat is consistent with measured 
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ability, past achievements, expressed and measured Interests, measured 
physical abilities and limitations and expressed values (Gysbers and Moore, 
1971). 

1.A2 To formulate a career plan which will provide fulfillment of^^pl rat Ions, 
personal needs , values, and lifestyle preferences (Wellman, 1971). 

1.^3 To Identify successes and failures within Individual's educational 
program. 

1.44 * 

PROCESS OBJECTIVES AND STRATEGIES: 

1*41 Provide opportunity for cooperative work experience* 

1.42 Provide opportunity for occupational visits. 

1.43 Provide writing opportunity to relate work to life style (A Kr-12 Guide/ 
Wisconsin, 1971). 

1«44 Provide experiences with simulation learning games and role-playing 
(Wellman, 1971). 

OUTCOMES: 

1.41 Ability to estimate Interests cxid aptitudes in each of ten b,W£lc areas. 

1.42 Ability to list three or more school subjects related to eaich of ten 
basic areas of interests and aptitudes. 

1.43 Ability to list three or more vocations and/or Jobs related to ten basic 
areas of interests and aptitudes. 

1*44 Estimate of interests is ^congruent with estimate of aptitudes in each of 

ten basic areas. 
EVALUATION: 

1.41 Evaluate individual responses with outcome criteria for the three levels 
of learning (Wellman, 1971). 

1.42 Establish congruency between individual and teacher-counselor identifica- 
tion of success and failure in the program. 
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2.0 GOAL: CAREER AWARENESS. To develop awareness of the sociological, 
economical, and psychological aspects of the world of work. Achievement 
of the goal through developmental progress and utilization of occupa- 
tional knowledge in formulation of career pattern In terms of potential 
satisfaction (Wellman, 1971). 

LEVEL K-3 

DEVELOPMENTAL OBJECTIVE: 

2.1 To develop an awareness of the necessity/desirability of work. 
BEHAVIORAL OBJECTIVES:' 

2.11 To describe necessity for work In society and Its desirability (Wellman, 
1971). 

2.12 To describe the consequences of someone not choosing their work- 

2.13 To name workers the Individual Is dependent upon. 

2.14 * 

PROCESS OBJECTIVES AND STRATEGIES: 

2.11 Discuss goods and services itsed by inalvldual's family. 

2.12 Discuss occupations of parent^ relative, or friend. 

2.13 Discuss occupations of teachers and 'ay they need students. 

2.14 Schedule outside speakers for the class to discuss what chey do fbr 
society. 

OUTCOMES: 

2.11 Ability to list reasons why people work (Wellman, 1971). 

2.12 Ability to name range of occupational clusters. 

2.13 Ability to name services provldea to society by outRlde spaakers. 

2.14 * 
EVALUATION: 

2.11 Evaluai:e e/itablished behavioral objectives. 

2.12 Establish four or five cltxsters as criteria standard. 
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2*13 Use criteria standards of teacher for services of outside speakers. 
LEVEL A-6 

DEVELOPMENTAL OBJECTIVE 

2.2 To develop an awareness of how occupational clusters differ. 
BEHAVIORAt OBJECTIVES: 

2.21 To describe a variety of occupational clusters. 

2.22 To distinguish goods and services In Industries. 

2.23 To list occupations required by the community. 

2.24 * 

PROCESS OBJECTIVES AND STRATEGIES: 

2.21 Use Illustrative text material to point out various job aspects. 

2.22 Have Individual divide parents' occupations Into goods and services 
Industries. 

2.23 Have individuals Invite city manager or equivalent to E|peak with them. 

2.24 * 
OUTCOMES : 

2.21 Increased knowledge of job requirements, characteristics, and rewards 
of a variety of clusters (Wellman, 1971). 

2.22 Ability to describe differences between goods production and service , 
Industries. 

2.23 Ability to. In writing, give reasons why certain occupations are required 
by th® community. 

evaluation: 

2.21 Establish criteria standard, five-cluster minimum. 

2.22 Establish performance level categorization: 5-good, 4-falr, 3-mlnlmiim. 

2.23 * 
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1-EVBL 7-9 

DEVELOPMENTAL OBJECTIVE: 

2.3 To develop awareness of values related to self and environment related 

to careers. 
BEHAVIORAL OBJECTIVES: 

2.31 To recognize certain sociological, econo^tlc, and psychological aspects 
of society and the influence upon careers (Wellman, 1971). 

2.32 To recognize a wide variety of occupations which may be classified 
several ways (A K-12 Guide/Wisconsin, 1971). 

2.33 To recognize supply and demand has an Influence on career opportunity 
(A K-12 Guide/Wisconsin, 1971). 

2.34 * ' 
PROCESS OBJECTIVES AND STRATEGIES: 

2.31 Provide experiences with simulation learning "Career Game." 

2.32 Provide role-playing opportunity. 

2.33 Use interview technique related to Job visitations. 
OUTCOMES: 

2.31 Increased knowledge of relationships between psychological, sociological, 
and economic aspects of careers (Wellman, 1971). 

2.32 Increased knowledge of relationships -between occupation and career decisions. 

2.33 Ability to demonstrate employment trends in two occupational clusters due 

to supply and demand. % 

2.34 * 
EVALUATION: 

2.31 Establish pre- and post-test differences.? 

2.32 Use of criteria set by teacher listing relationship of psychological, 
sociological, and economic factors to careers. 

2.33* 
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LEVEL 10-12 

DEVELOWIENTAL OBJECTIVE: 

2.4 To develop awareness of personal career alternatives. 
BEHAVIORAL OBJECTIVES: 

2.41 To Identify a broad range of career options as personal alternatives to 
set tentative goals in one or more occupational areas (Wellman, 1971). 

2.42 To recognize the need to be flexible to meet Job characteristics 
(A K-12 Guide/Wisconsin). 

2.43 To list areas of possible compromise when relating career options to 
personal needs and goals. 

2.44 * 

PROCESS OBJECTIVES AND STRATEGIES: 

2.41 Provide experience with Holland's theory /classification (Wellman, 1971) 

2.42 Provide participation in field trips and Job interviews. 

2.43 Provide work experience when possible related Co occupational goals. 
• 2.44 * 

OUTCOMES: 

2.41 Ability to express commitment to career goals (Wellman, 1971). 

2.42 Ability to write an outline of sequential goals*. 

2.43 Ability to write an outline of career alternatives in rank order. 

2.44 * ' 
EVALUATION: 

2.41 Establish criteria standard related to number of occupational cluster 
altetnatives. 

2.42 Compare individual with teacher/counselor estimations of reality. 

2.43 * 
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3.0 GOAL: CAREER PLANNING. For individuals to develop planning skills in 
relationship to career, education, and personal preparation. The Indi- 
vidiial will become aware of the relationships of career, education, and 
personal style and how this interaction affects career planning. 

LEVEL K-3 

DEVELOPMENTAL OBJECTIVE: 

3.1 To be aware of the process of planning (Gysbers and Moore, 1971). 
BEHAVIORAL OBJECTIVES: 

3.11 To demonstrate an understanding of the usefulness of planning. 

3.12 To identify situations requiring decisions. 

3.13 To identify situations .resulting in a change of decision. 

3.14 * . , 
PROCESS OBJECTIVES AN13 STRATEGIES: 

3.11 Have individual plan own appropriate school dress. 

3.12 tlave individual participate in a class decision. 

3.13 Have individual describe verbally why he/she changed a decision. 

3.14 * 

OUTCOMES: 

3.11 Ability to identify frqm a list of activities those requiring planning 
^ (Gysbers and Moore^ 1971) . 

3.12 Ability to describe how the class makes a decision. 

3.13 Ability to list reasons why a decision might be changed. 

3.14 * 

EVALUATION: ' • 

3.11 Establish criteria percentile standard. 

3.12 Have individual list two events which led class to a decision. 

3.13 Have individual describe verbally why he/she has changed a decision. 

3.14 * 
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OUTCOMES: 

3.11 Ability to Identify from a list of activities those requiring 
planning (Gysbers and Moore, 1971). 

3.12 Ability to describe how the class makes a decision. 

3.13 Ability to list reasons why a decision might be changer^. 
3.1A * 

EVALUATION: 

3.11 Establish criteria percentile standard. 

3.12 Have Individual list two events which led the class to a decision. 

3.13 Have Individual list two reasons why a decision might be changed 
(teacher criteria for performance). 

LEVEL 4-6 

DEVEtOPMENTAL OBJECTIVE? 

3.2 To understand responsibility for planning career. 

■ 

BEHAVIORAL OBJECTIVES: 

3.21 To evaluate environmental Influence upon one's planning. 

3.22 To evaluate one's characteristics In relation to planning. 

3.23 To Identify how past decisions have led to present decisions. 

3.24 * 

PROCESS OBJECTIVES AND STRATEGIES: 

3.21 Have Individual discuss situations that have Influenced decisions; I.e., 
teachers, parents, money. 

3.22 Have Individual discuss personal characteristics that relate to decisions she/ 
he has made. 

3.23 Have Individual discuss consequence of past decisions. 

3.24 * 
OUTCOMES: 

3.21 Ability to participate effectively In simulation employment office 
(Gysbers and Moore, 1971). 
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3.22 Ability to discuss with the counselor personal characteristics and how 
they relate to decisions. ^ 

3.23 Ability to list consequences of past decisions. 

3.24 * • ^ r . 
EVALUATION: 

3.21 Establish pre- and posttest differences. 

3.22 Use performance level criterion set by teacher. 

3.23 * 
LEVEL 7-9 

DEVELOPMENTAL OBJECTIVE: 

3.3 To understand the relationship between academic choices and career 

planning. 
BEHAVIORAL OBJECTIVES: 

3.31 To choose a curriculum plan related to tentative career choice. 

3.32 to describt how self-interests relate to career choice. 

3.33 To identify possible future changes in chosen career and/or curriculum. 

3.34 * 

PROCESS OBJECTIVES AND STRATEGIES: 

3.31 Have individtoal Interview staff members. 

3.32 Have individual participate in small group discussion. 

3.33 Have individual conduct field trip intervie^^, discussing the changes 
present employees have made. 

3.34 * 
OUTCOMES: 

3.31 Ability to choose elective courses through mass registration. 

3.32 Ability to state which courses most directly relate to a preferred career. 

3.33 Ability to list possible alternative careers in relation to a chosen 
curriculum. «i' . 66 

3.34 * 
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EVALUATION: 

3.31 Compare student and teacher/counselor estimation of reality. 

3.32 Use of teacher criterion of ability to list alternatives within the 
curriculum. 

3.33 * u 
LEVEL 10-12 

DEVELOPMENTAIr> OBJECTIVE : 

3.4 To Identify alternatives In career planning (Gysbers and Moore, 1971). 
BEHAVIORAL OBJECTIVES: 

3.41 To evaluate alternative careers. 

3.42 To choose a hypothetical alternative goal and reformulate plans. 

3.43 To Identify situations In the community that could effect career planning. 

3.44 * 

PROCESS OBJECTIVES AND STRATEGIES ^ 

3.41 Provide job visitations. 

3.42 Have individual research community Information and situations 
affecting his/her career. 

3.43 Provide Job Interviews In chosen career cluster. 

3.44 * ^ . 
OUTCOMES: * ^ . 

3.41 Ability to evaluate and rank order five alternative clusters related 
to own potential. 

3.42 Increased knowledge of community's economy and other career related 
variables. 

3.43 Ability to complete three interviews in the community. 

3.44 * 
EVALUATION: 

3.41 Determine If the Individual's listing of career clusters and potential 
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l6 congruent with the perceptl-n of the couna elor /teacher . 
3.A2 Have Individual evaluate own interviews aa successful or nonsuccessf ul 

in terms of career goals* 
3.43 * 

4.0 GOAL: CAREER PREPARATION. To develop understanding of the relationships 
between academic work and career preparation and to progress developmen- 
tally in personal career preparation. 

LEVEL K-3 

DEVELOPMENTAL CBJECTIVE: 

4.1 To develop awareness of academic skills. 
BEHAVIORAL OBJECTIVES: 

4.11 To identify academic skills needed in the world of work (Wellman, 1971). 

4.12 To identify communication skills needed in work (Gysbers and Moore, 1971). 

4.13 to identify psychomotor skills needed in work (A K-12 Guide/Wisconsin, 1971) 

4.14 * 

PROCESS OBJECTIVES AND STRATEGIES: 

4.li Have individual discuss educational level of parents. 
4.ir: Consult CI.S. (Career Information System) (CCEM, 1971). 

4.13 Have individual discuss psychomotor skills needed in school-related 
work or parents' work. 

4.14 * 
OUTCOMES: 

4.11 Ability to list three valid relationships between various disciplines 
and occupation'^, 

4.12 Ability to identify workers who rely on language in their occupation. 

4.13 Ability to list skills needed to do school assignments. 

4.14 * 
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evaluation: 

4.11 Have Individual participate In discussions. 

4.12 Have Individual comnunlcate through verbalizing thoughts. 

4.13 Have Individual name and demonstraxe a skill related to school work. 
LEVEL 4-6 . 

DEVELOPMENTAL OBJECTIVE: 

4 

4*2 To be aware of good and bad emplo^rment characteristics* 

r 

BEHAVIORAL OBJECTIVES: — ' 

4.21 Identify employee characteristics taken Into consideration by employers 
(Wellman, 1971). 

4.22 Identify employee characteristics considered positive. 

4.23 Identify employee characteristics considered negative. 

4.24 * 

PROCESS OBJECTIVES AND STRATEGIES: 

4.21 Provide Job visitation opportunity. 

4.22 Schedule Career Day source speakers. 

4.23 Have Individual choose an occupation and Interview a member. 

4.24 * 
OUTCOMES: 

4.21 Ability to name employee characteristics both favorable and unfavorable. 

4.22 Ability to list positive and ne^tlve characteristics for Job Interview 
choice. 

4.23 Ability to relate three school characteristics to three employment 
characteristics. 

4.24 * 
EVALUATION : 

Establish minimum performance level criterion as follows: 

4.21 Have individual list five general employee characteristics both favor- 
able and unfavorable. 

6P 



4.22 Have individual relate three school characteristics to three employee 
characteristics for at least two cluster areas. 

4.23 Have individual list five specific employee characteristics for Job 
' interview cl^oice. 

4.24 * 
LEVEL 7-9 

DEVELOPMENTAL OBJECTIVE: 

4.3 To be aware of placement procedures both (educational and occupational. 
E HAVIORAL OBJECTIVES: 

4.31 To describe how and where to seek general employment. 

4.32 To describe the secondary curricula choices. 

4.33 To describe the educational level needed for various occupations. 

4.34 * 

PROCESS OBJECTIVES AND STRATEGIES: 

4.31 Have individual take part in simulation learning and role playing 
of Job/ school interviews. 

4.32 Provide presentation of educational levels. 

4.33 Have the individual give written and oral presentations of various 
occupations and required educational levels. 

4.34 * 

OUTCOMES: 

4.31 Ability to complete a written application for employment. 

4.32 Ability to complete a written s ^ (*nle for a curricular option 
(Wellraan, 1971). 

4.33 Ability to list three curricular options. 

4.34 * 
EVALUATION: 

4.31 Have individual hand in successfully completed employment application. 
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A. 32 Have individual hand in successfixlly completed curricular schedule. 
A. 33 Have individual recite orally an occupation appropriate for each secon 

dary curricula option. 
A.3A * 
LEVEL 10-12 

DEVELOPMENTAL OBJECTIVE: 

A. A To be aware of Job and educational alternative potential placement. 
BEHAVIORAL OBJECTIVES: 

A.Al To demonstrate skills needed for employment upon completion of voca- 
tional curriculum (Wellman, 1971). 

A.A2 To demonstrate skills needed for college placement upon completion 
of pre-college curriculum. 

A. A3 To demonstrate skills needed for further occupational education upon 
completion of a vocational technical curriculum. 

A. 44 * 

PROCESS OBJECTIVES AND STRATEGIES: 

4.41 Provide psychomotor training and experience in voc-tech aspects. 

4.42 Provide specific cognitive training in academic disciplines. 

4.43 Provide guidance counselor presentation of job and educational 
alternatives. 

4.44 * 
OUTCOMES: 

4.41 Ability to attain competency rating by instructor or employment 
placement. 

4.42 Ability to successfully complete Washington Pre-College Test. 

4.43 Graduation with 2.5 CPA or above. 

4.44 * 
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EVALUATION: 

4.41 Criteria standards of voc-tech competencies are established. 

4.42 College admission standards are established. 

4.43 * , ^ 

5.0 GOAL: WORK AND LEISURE. For Individuals to recognize that personal 
satisfaction In a career Is related to Involvement In leisure time 
activities (Gysbers and Moore, 1971). 

LEVEL K-3 

DEVELOPMENTAL OBJECflVE: 

5.1 To und^stand the function of leisure time and life style (Gysbers and 
Moore, 1971). 

BEHAVIORAL OBJECTIVES: 

5.11 To demonstrate awareness of the difference between work and leisure 
(Gysbers and Moore, 1971). . ' 

5.12 To demonstrate awareness of times during the school day considered 
leisure. 

5.13 To demonstrate awareness of parent's way of using leisure time. 

5.14 * 

PROCESS OBJECTIVES AND STRATEGIES: 

5.11 Have Individual experience listening and observing presented leisure 
and work activities. 

5.12 Have Individual consult C.I.S. (Career Information System) (CCEM, 1971). 

5.13 Have individual participate In a dlscubislon of fun activities such as 
games, sports, and hobbles. 

5.14 * 
OUTCOMES: 

5.11 Ability to designate from a list of activities the nature of work 
and leisure. n c% 
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5. 12 Ability to present orally a definition of leisure. 

5.13 Ability to recite orally a definition of life style. 

5.14 * 
EVALUATION : 

5.11 Have individxial give an example of a school or home work activity. 

5.12 Have individual give an examplr of a school or home leisure activity. 

5.13 * 
LEVEL 4-6 

DEVELOPMENTAL OBJECTIVE: 

5.2 To understand the funct:^on of leisure time and life style. 
BEHAVIORAL OBJECTIVES: 

5.21 To understand the importance of leisure time enjoyment (Gysbers and 
Moore, 1971). 

5.22 To understand the differences among various life styles. 

5.23 To understand that leisure means different things to different people. 

5.24 To understand ways that occupations influence life style. 
PROCESS OBJECTIVE.S AND STRATEGIES: 

5.21 Have individual experience a leisure time field trip, i.e., camp out. 

5.22 Have individual share and discuss various individual leisure interests. 

5.23 Schedule a resource speaker from the city recreation program presentation. 

5.24 * 

OUTCOMES: 

5.21 Ability to list a number of leisure interests and explain reasons for 
enjoyment. ( 

5.22 Ability to list why leisure means different things to different people. 

5.23 Ability to list why occupations influence life style. 

5.24 * 
EVALUATION: 

5.21 Have individual successfully accomplish choice of one specific occupation 



and list three life style Influences. 

5.22 Have Individual successfully accomplish choice ot one leisure activity 
and explain three reasona for enjoyment. 

5.23 * 
LEVEL 7-9 

DEVELOPMENTAL OBJECTIVE: 

5.3 To understand the function of leisure time and life style. 
BEHAVIORAL OBJECTIVES: 

5.31 To understand that career choice affects the amoxint and type of leisure 
time (Gysbers and Moore, 1971). 

5.32 To understand the relationship between salary and type of life style 
(Gysbers and Moore, 1971). 

5.33 To understand the various methods of classifying occupations (Gysbers 
and Moore, 1971). 

5.34 * 

PROCESS OBJECTIVES AND STRATEGIES: 

5.31 Have individual explore work experiences and interview. 

5.32 Have individual read assignments from Occupational Handbook. 

5.33 Provide guidance presentation of attitudes/interests and discussion. 

5. 34 * 

i 

OUTCOMES: 

5.31 Ability to identify a number of careers as to the amount of time and 
responsibility. 

5.32 Ability to list five valid ways occupational clusters are related to 
life style. 

5.33 Ability to list occupational clusters. 

5.34 * 
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EVALUATION: 

5.31 Have Individual establish minimum criteria standard at the 75% level. 

5.32 Have Individual lis five career clusters. 

5.33 Have Individual list five ways career clusters are related to life style. 

5.34 * 
LEVEL 10-12 

DEVELOPMENTAt OBJECTIVE: 

5.4 To undei^stand the function of leisure time and life style. 
BEHAVIORAL OBJECTIVES: 

5.41 To make tentative career plans vd.th individual application of one's 
ovm values concerning leisure time. (Gysbers and Moore, 1971). 

5.42 To describe preferred life style. 

5.43 To identify values and needs met by leisure activities. 

5.44 * 

PROCESS OBJECTIVES AND STRATEGIES: 

5.41 Have individual study occupational time and responsibtlity obligations. 

5.42 Provide Sroup discussion of values, needs^^, and leisure. 

5.43 Schedule counselor presentation of personality need and value systems. 

5.44 * 

OUTCOMES: 

5.41 Ability to match and contrast various occupational obligations in relation 
to own values and formulate a tentative plan of action. 

5.42 Ability to match life styles \rixi\ probable value and needs. 

5.43 Ability to rank order a list of terminal values* 

5.44 * 
EVALUATION : 

5.41 Provide discussion of congruence between individual and teacher/counselor 
perceptions of aspired occupational choice on a satisfactory-unsatisfactory 
basis. 
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5.42 Provide discussion of congruence between Individual and teacher/cpuhBelor 
perceptions of aspired occupational choice on a satlsfactory-unsatlBfactory 
basis. 

5.43 * 



References: 

A K-12 Guide for Integrating Career Development Into Local Curriculum . Depart- 
ment of Public Instruction, State of Wisconsin, Madison, WI, 1971. 

Comprehensive Career Education Model . Columbus, OH: Center for Vocational 
and Technical Education, 1971. 

"Elements of a Quality Career Guidance, Counseling, and Placement Program." 
Coordinating Council for Occupational Education, Olympla^ WA, 1975. 

Gysbers and Moore. Career Conscious Individual . Columbia, MO: University 
of Missouri, 1971. 

Wellman, Frank E. Systems Model for Guidance Program Development and Evaluation . 
Paper presented at Career Development Guidance, Counseling and Placement 
Project, National Training Conference, St. Louis, MO, 1971. 
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Learning Activity 4 -b 

Select an area of vocational educatlon-**a class, project 
or program — and devise a list of 20 criteria, 10 for each 
learner activity , and 10 for program outcomes, which could 
se^^rve as a basis for evaluation of a project, program or 
class. V 



References : 

Superlntendei^t of Public Instruction. A Manual for Project 
Applicants and Grants . Title III, Elementary and 
SecondaiT Education Act In Washington State. Olympla, WA: 
Superintendent of Public Instruction, 1971. 

Wasdyke, R. G. , and Murphy, R. R. "Form for Evaluating 

Objectives In the Occupational Area." Occupational Perform- 
ance Objectives . Princeton, NJ: Educational Testing 
Service, 1974. 
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CRITERIA FOR EVALUATING CURRICULUM GUIDE 
Currlculixm Guide Evaluated 



CONTAINED IN GUIDE 

CRITERIA 

Yes No/Not Needs Im- 
Included provement 



I. PROGRAM GOALS: 



Consistent with needs (student and employinent) 








Beh&vloral oblectlves designed to meet seals 








Consistent with level and abilities of students 








Relevant attitudes. knowledRe, and skills to Jobs 








Feasible within time and budget 








Statements clear and understandable to students, 
teachers » administrators 








Establishes learning directions and emphases 








Consistent with stated philosophy 








Consistent wltn available resources 








Describes terminal behavior 









II. CONTENT (tlurrlculum) : 



Clearly relates to course goals 








Valid up-to-date materials 








Includes general principles as opposed to facts 









ril. LEARNIKG ACTIVITIES: 



Safe with adequate supervision 








Effective for meeting objectives 








Variety for student Interest 








Balance in modes of learning 








Flexible to meet student needs 








Consistent with "real world" activities 








Organized in logical progression 








Flexible in school setting 









IV. LEARNING MATERIALS: 



Available in sufficient quantity 








Instrumental in achieving behavioral objectives 








Meets capabilities of students 








Quality consistent with price 








Suitable for various teaching methods 








References and resources Included 









V. EVALUATION: ^ 



Provision made for OJT follow-up 








Type compatible with behavioral objectives 








Criterion referenced and norm referenced testing 
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CONTAINED IN GUIDE 

CRITERIA 

Yes No/Mot Needs Im- 
Included provement 



Provides for student feedback 








Includes pre-testlnK and initial screening 








DlstlnK'^lshe.'i varlbus levels of IdiovledKe 








Provides for program evaluation and revision 








Evaluation technique contributes to learning 









I 

1 
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NAME OF PROJECT PROPOSAL: 

at 



REVIEWER APPLICANT 
!• Brief abstract or description of project: write concisely In a few 
sentences highlighting the central objectives, the target population, 
procedures and activities, and evaluation. Do hot attempt to detail 
each part of. the project, but capture In abstract form the central 
features. 

II. Needs Low High 

1 2 3 4 5 

Degree to which : 

1. needs are clearly stated () ()() () () 

2. need statements are specific 
enough to leiid real direction to 

project ()()()()() 

3. need statements make reference 

to behavior of learner () () () () () 

COMMENTS: 



III.- Objectives 



^ Degree to which : 

> 

1. N, objectives are functionally 

\ J^related to stated needs ()()()()() 

2. objectives are assessable at 

relatively frequent Intervals () () () () () 

3^ objectives Include description of: ' 

a. performance being observed () () () () () 

b. conditions under which behavior 

Is to occur ()()()()() 

c. criterion of acceptablle perfor- 
mance Including time dimension, 

accuracy, and proficiency level () () () () () 
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Low High 

12 3 4 5 

4. objectives Indicate changes In 

student performance () () () () () 

5. staff objectives are specified () () () () () 
COMMENTS: 



lV. Procedures and Activities 
Degree to which ; 

1. logical progression ofevents 

Is evident ^ ()()()()() 

2. the responslblllt;; Tor procedural 

steps Is evident ()()()()() 

3. procedures are clearly enough 
described so that an outside reader 
could Implement those procedures 

in very similar form ( ) " ( ) ( ) ( ) ( ) 

4. there are specific procedures 

^or each objective ( ) ( ) ( ) ( ) ( ) 

5. procedures are described to 
monitor staff activities and 

responsibilities ( ) ( ; ( ) ( ) ( ) 

6. procedures lend themselves to 

continuous evaluation () () () () () 

. . ' f 

7. procedures Indicate how evaluation 

Is going to be Incorporated In * 

on-going program operation () () () () () 

8. procedures are capable of being 
exported ''and Implemented ±n ' \ 

another district ()()()(>() 

9. the sequence of events Is 

delineated (timelines) ( ) ( ) ( ) ( ) ( ) 

V 

10. evidence Is given that facilities, 
materials » and resources are 

available to accomplish objectives ( ) (^.^( ) ( ) ( ) 

11. proposed activities are educa- ^ ^ 
tlonally sound and logically . ^ 

related to objectives ()()(') ( ) ( ) 



81, 



Low High 
1 2 3 4 5 

12. activities are Innovative and 

seem likely to effectively motivate 

student learning () ()() () () 

COMMENTS: 

V. Evaluation 
• Degree to which : 

1. evaluation design Is clearly 

described ( ) ( ) ( ) ( ) ( ) 

2. evaluation data are derived 
from valid and reliable 

Instruments and techniques ( ) () () () () 

3. evaluation Incorporates 
measures of each specified 

objective ()()()()() 

4. evaluation data measure progress 
tovard objectives, not Jtist whether 

objectives were or were not achieved () () () () () 

5. provisions are made for modifica- 
tion of procedures based on feed- 
back from on-going evaluation () () () () () 

6. evaluation process will be under- 
standable to appropriate audiences () () () () () 



CWIMENTS: 



VI, Coordination and management of learning activities: 

1. staff and management respon- 
sibilities, are clearly identified () () () () () 

2, coordination with total curriculum 
is evident - that is, thero. sc^enuB 
Lo be appropriate linkage with 

total school program () () () () () 

COMMENTS: 
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VII. Strengths and weaknesses of project: 
Strengths 



Weaknesses 



Your comment on educational signif ic£<nce of project: 



VIII • Recommendations : 

If you were reviewing the proposal for a funding agency or a school 
district would you: 



reconnend project 




high priority level 




medium priority level 



low priority level 



not recommend project 




recommend with the 



following revisions: 
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CATALOG AREA 
BY SOURCE 



FORM FOR EVAL. A 'Pii? OBJECTIVES IN 
THE OCCIPATIG AREA 



PERFORMANCE OBJECTIVES JOB LEVEL 



GRADE LEVEL 



METHOD OF 
VALIDATION 
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PERF0RI4ANCE OBJECTIVE 5 



Specify the sources of error and evaluate the criteria used In measures which 
determine student outcomes In vocational programs. 

In addition to paper and pencil testing rnd Informal observation, 
systematic observation of both the learning process and , roducts provides useful 
evaluation data. Such observations can be those of a teacher or supervisor, or 
learners themselves may observe and rate the product of their efforts. 

Numerous observational approaches have been suggested ranging from Informal 
instructional observation to highly structured and sophisticate systems, which 
require considerable training to use. 

For most teachers and supervisors, falrW reliable data may be obtained 
If a ^ew general principles are observed: 

1. Observations should be made In representative situations. 

2. Careful attention must be paid to definition of behaviors, details 
of process, or specific criteria when evaluating a product. 

3. The process or observation should not Interfere with the leamln^r 
process or behavior being observed so as to render the observation 
atypical. 

The following cautions or obstacles to effev.t:lve use of observa Ion 
shculd a ' 3o be considered: 



High degrees of reliability or consiF^ency are difficult to 
obtain even when the observer Is highly trained, that Is, c servatlon 
contains a hlgl. degree of subjectivity. The best antidote Is to 
have a careful knowledge c£ what to observa. and to define the 
characteristics as specifically C3 possible. General observations 
of an unspecified nature are usually worthless. 




2. Many characteristics, particularly be:isvloral characteristics as 
usually labeled, are ambiguous and produce a great variation In the 
Judgments ude about them. Such qualities as leadership , self- 
confidence, good citizenship, v^cc, serve as examples. In order to 
produ^a nx>re consistent Interpretations of such qualities, specific 
observable beha/loral ch-^ acterlstlcs associated with more general 
labels should be Identified. For example, ''What specific types of 
behavior do students demonstrate wuen they are showing leadership 
qualities?" or "How do students behave when they are being good 
citizens?" 

3. The error of leniency Is common among raters and must be guarded 
against. Resr arch has shown that observers tend to rate high thoae 
persons with whom they are well acquainted or ego-Involved. In 
ar'uitiou to the clear definition of that which Is to be observed, 
id advised that any observer should ask other knowledgeable persons 

to check Oil their observations periodically to detect bias of any sort. 

4. Likewise the halo effect or error of generalization Is commqn In 
making observations. This means that persons who formulate a set 
or 5l:ancc " v-rards t>:e quality of that being observed or rated, 

tend to racf ^'.7 parte of the process, product, or behavior In accord 

with that gc ieiai set. Tralr^xng In observation will help observers 

>ecomf. more dlscrlml iating.. 

Selected Ref ereiices-Obser/aticn: 

in, Amol'' J. "Construction and Use of Obsf^rvatloiial Tools." T'leasurement 
a nd E v eluatloii of Leamlny >. DuJuque, lA: "m. C. '4rown, 19tT. 

Chcy. tef "S 

Michaels ard Karnes. Measuring Educational Achievement . New York: McGraw Hill, 
1950. Chapter 13*. 

Ttenbrink, T erry JP. ^Evalugtlotix A-Practlcal Guide for Teac h ers . New York: 
McGraw Hill, 1974. Chapter 6. 
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Teat Development Suiff* I natrtanent Gulde> Princeton, NJ: Educational 
Testing r. e, 1971. 



Instm^ 5 lo Objec t ive 5«1 

The lef ' will be able to construct observation sheets and test 
Items, and apply the appropriate criteria for the evaluation of each. 
Learning Activity 5.1-a 

1. Create an observational rating scale or ch> OLlst In a 
vocational area of your choice which nforms to the 
following specifications: 

a. Is In accord wl*^h prior knowledge of what to observe, 

b. Is In accord with specific objectives of learning 
activity, or purposes of observation, 

c. est-Mlshes clear definition of characteristics to 
be observed, 

d* includes 3-5 levels of proficiency or gradations of 
characterls tics , 

e. provides for sunmary statements and understandable 
profile, 

f. has clear dlractlons and is free from ofl^nslve 
qualities, 

g. ir eludes time lines, frequency, and context. 

2* Select wo or three examples of observation sheets and 
critique them for content, ap^^roprlatcness to proj^ram 
objectives, prof: :iency levels, and validity. 

3. Select two or three sheets anc evaluate them using test 
construction c it^ria. 

Reference: 

Peatman, J.G. "Definition and Sources of Error in Saatpling." 
Descripti/e and Satrpling Statistics . New York: Harper, 

88 



79 



Test consrrtictora and eyaluators often refer to tablea which corfaln 
a jystemttClc listing of objectives, reflecting the relative emphasis on each 
as a table of specifications. Such tablea become the blueprint for the 
measurement and evaluation Instruments. They may be simple or complex, depen- 
ding on the number of objectives jntalned In a unit. A few examples are pro- 
vided. The Importance of the relationships between testing and program objec- 
tive Is essential If valid evaluation Is to occur. The following represents 
criteria fc^r rel^itlng tests to objectives. 

1. Performance Objectives - statements which say very precisely what be- 
havl -ral changes In the learner are expected to occur as a result of the 
Instruction given, must be written and the particular Pleasures developed 
to assess those behaviors* 

2. Each learner pet-rcrraance objective must Inclvae the following: 

A - (audience) The learners have been described. 

*B - (behavior) The behavior to be exhibited to the 

product to be developed is cJeecrlbed. 

*C - (condition^ ) '^e conditions under which the behavior 

1.1 to occur or the 'product Is to be 
devr^ 3ped are described. 

. - (/ . •'^^e) me ':rlterla for determining uhe 

acceptability of the behavior or product 
are describt^'d. 

E - (fcvaluatlon) The evaluation processes are described. 



*Api Jefintd in the Module, "Developing Curriculum: Goals, Objectives, 
and InL cvMCv'lcnal Plans," B, C, and D ere the essential e'^'^ents of a 
performan objective. 
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Use of Criteria for Analysing Objectives 



CRITERIA 


OBJECTIVE 
A 


OBJECTIVE 
B 


O^ECTIVE 
C 


Oi^JECTIVE 
D 


Who 


Student (pu- Ll, 


Pupil 


Learner 


Learner 




learner) 








What behavior 


will bui 


will bake 


will chair - com- 


will type 








mittee to develop 




What result. 


a cedar chest 


a cake 


a student conduct 


a letter 


product or 






code 




per fcrmance 










Under what 


during schov < 


at hpme during 


wif.thin a one month 


in a 5-minute 


conditions 


day In shop 


weekend 


period 


timed teat 




using tools 










and materials 








What standards 


Chest roust 


Cake must be 


which is ratified by 


4 1 an average 




sell at school 


Judged acceptable 


majority vote of 


speed of 35 




contracted 


In texture, taste 


stud^at body 


words per 




price to 


appear an . by 




minute with a 




furniture 


majority of 




minimum of 




store 


classmat**'" 




5 errors 



A wide VBT'e^v of standardised tests are available in the basic skill 
aveac and a fe^r ^ther standard techniques in vocational educallon. These; 
will be listed id crirlqued in the various editions of Buros, Mental 
Measurement Yearbook * 
References ^ Testing: 

Bloom» B.S. ; Hastings^ J.T. ; anr; M^dans, G.F. Handbook on Formative and 

Summatlve Evaluation of Stu dent Learning . New York: McGraw Hill, 1971. 

Euros, Oscar K. Mental Measurement Ye a rbook , rilghlatid P^^rks, NJ: Gyphron 
Press, 1970. 

Green, J >hn A. Teacher-ma ^ Tests , 2nd ed. New York: Harper and Row, 1975. 

M^iger, Robert. Preparing Instructional Objectives . Belmont, CA: i^eaiom 
Publishers, 1975. 

o 

Payne, David A. The Assessment of Learning: Cognitive and \ffective . 
Lexington, MA: D.C. Heath and Company, 1974. Parts II and III. 
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Progress Chart - Manipulative Operation 



Grade 
Course 



Period 



Instructor 



Code* 



1. Excellent 

2. Good 

3. Fair 

4. Poor 



1. Unsatisfactory 

2. Aver i r. 

3. Excellent 




Perform a blood 
pressure reading 



Operate oxy-acetylene 
welding equipment 



3. Cut a sheer of metal 
with snips 



Proficiency levels 



Excellent !• 



Good 2. 



Average-Fair 3, 



Poor/Unsatis- 

f.u ory 4. 



Followed prescribed procedure in detail - accurate and correct. 
Obse ved safety procedures. Used all tools and equipment 
correctly. Completed operation in minimum lenj?:**^ of time. After 
initial Instruction, required no assisti.-ice. 

Followed prescribed procedure. Work of high quality. Observed 
safety precautions. Completed operation in average time> Re- 
quired some help irom instructor. 

Had difficulty iu following correct procedure. Obsf rved only 
obvious safety precautions. Work of poor quality. Required 
maximum time. Required much help. 



Was unable to follow correct procedure. Failed to complete 
work in time. Failed to i 3e safety precaution. Misused tools 
and equipment. Did not respond to help i$iven. 



* 1 codes represent two alternative -methods for rating; either may be used. 



Progress Chart - Jobs/Assignments 



Grade Period 
Course 


Code 
X » Complete 
0 « Incomplete 






1. Butt Weld 












2. T Weld 












3, Tilet T. Weld 













Proficiency level 

X " Weld free from slog pockets, uniform width, withstands 
1000 PSI pressure. 

0 any of the following: Slog pockets, non-uniform width, 
fails at 1000 PSI pressure. 



\ 



■A 
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EMPLOYEE EVALUATION FORM 
Name: Date: 



Department : Job Title : 

PURPOSES OF THIS EMPLOYEE EVALUATION: 

To take a perbonal Inventory, to pinpoint weaknesses and strengths, and to out- 
line a i agree upon a practical Improvement program. Periodically conducted, 
these evaluations will provide m history of development and progress* 
INSTRUCTIONS: 

Listed below are a number of traits, abilities, and characteristics that are 
Important for success In business. On each rating scale, place an "X" over 
the descriptive phrase which moat nearly describes the person being rated. 
(If this form Is being used for self-evaluation, you will be describing your- 
self.) '-^ 

Carefully evaluate each of the qualities separately. 

Two common mistakes in rating are: (1) a tendency lo rate nearly everyone as 
"average'* on every trait instead of being raore critical in Judgment. The 
rater should use the ends of the scale as well as the middle, and (2) the 
"halo effect;" i.e., a tendency to rate the same individual "excellent" on 

every trait or "poor" >n every trait based on the overall pictulre one has of 

V 

the person being luted. Howevet, each person has strong points and weak 
foints and these should be Indicated on the rating pcale. 

ACCURACY is the correctness or work duties perfor d. 

Makes frequent Careless; makes Usual!)^ accurate Requires little Requires absolute 
errors. recurrent errors makes only aver- supervision; is minimism of super- 
age number of exact and pre- vision; is almost 
mistakes else most of always accurate 

time 

Daniels, William E. Diversified Occupa*-i ons. Olympia, WA: Coordinating Council 
for Occupational Education, 1973. 
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STUDENT W •< EXPERIENCE 
FORM FOR 

EMPLOYER'S ESTIMATE OF COOPERATIVE TRAINEE 
Cooperative Trainee Date 



Conipan y ^Occupation 



Supervisor: This rating sheet Is used by the school work coordinator for 
counseling purposes. Please chec;^ (:» e column which best describes the 
student trainee. 

CHARACTERISTICS T^.XCELLENT GOOD AVERAGE ^AIR POOR 



Quality of work 



Attitude toward job 



Attitude toward fellow workers 



Dependability (on job assigned) 



On time and at work regularly 



Initiative (find work to do) 



Pertsonal appearance 



COMPARED TO A NEW YOUNG EMPLOYEE, WHAT GRADE WOULD YOU GIVE? 



Please feel free to list additional comments regarding the progress of the 
student belov or on the back of this form. 



Employer's Signature 
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ALERTNESS Is the ability to grasp Instructions, to meet changing conditions and to 
solve novel or problem situations* 



Slow to Requires more Grasps Ins true- U^Hilly quick Exceptionally 

"catch on" thdn average tlons with to understand keen and alert 

Instructions average ability . andj learn 

and explanation^:? 

'X 

CREATIVITY Is talent for having new Ideas, for finding new and better ways of doing 
thl' ?s, and for being Imag inative. i 



Rarely has a ' Occasionally 
new Idea; comes up with 

Is unlmaglna- new idea 
tlve 



Has average Frequently 

Imagination; has "Buggests aew 

re^sqnable' ways of doing 

numWr new things - Is 

Ideas / ' very Imaglna- 

S tlve 1 



Contli Jially seeks 
new ar better 
ways of doing 
things; Is ex- 
tremely Imagina- 
tive 



FRIENDLINESS Is the' sociability iamd varnfch ^ich an individual iL^^arts in hi 
attitude toward customers, other employees, supervitpra , and the persons he may 
supervise. A . y ' * / 



Very distant Approachable;^ 
and aloof friendly once 

known by otheA 



Warm, friendly. Very 'Sociable 
sociable , ' and otit-golng 



Extremely sociable 
excellent at estab 
llshlt >ood will 



PERSONALITY is an individual's behavior characteristics or her personal suitability for 
the job. . . ' 



Very juntidy. Sometimes untidy Generally neat 
poor 'taste in and careless and clean; 

dress about personal ' satisfactory 

?!ppearance personal 

appearance 



Careful about Unusually well 

personal , groomed; very 

appearance good neat ;' excellent 

taste in dress taste in dress 



PHYSICAL FITNESS is the ability to work consistently and with only moderate fatigue.^ 
(consider physical alertness and energy) 



Tires easily; F*requently tires Meets physical 
is weak and €md is slow and enetg> job 

frail I requirements 



Energetic; 
seldom tf.res 



Excellent^ health 
no fatigue 



ATT^lf^ANCE is faithfulness in coming Lo work dally and conforming to work hours. 



Often abc^ent 
wlfhout ,^ood 
e^r v. 'x^B d /o r 
f requeue ly 
reports for 
work late 



L|ax in attend- 
ance and/or 
reporting for 
work on time 



Usually present 
and on time 
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Very prompt ; 
regular in 
attendance . 



Always regular 
and prompt; vol- 
unteers for over- 
time when needed 
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HOUSEKEEPING Is the orderliness and cleanliness In which an Individual keeps his/her 
vork area. 



Dlsord^^rly or Some tendency to Ordinarily keep Quite consclen- Unusually neat, 
untidy be careless and vor': ar^.a fairly tlous about clean, and 

untidy 1^ icar neatness and orderly 

cleanliness 

DEPENDABIHTY Is the a lllty to do required Jobs well with a minimum of supervision. 



Requires close Sometimes re- 
sup e rvl s 1 on , r eq ul r e s 
Is unreliable prompting 



Usually takes 
cdre of neces- 
sary tasks and 
cbmpletes with 
reasonable 
promptness 



DRIVE Is the jde^slre to attain goals to achieve. 



Requires little 
supervision; 
Is reliable 



Requires abso- 
lute minimum 
of supervision 



I 



Has poorly df- Sets goals too 
' fined goalsV. low; puts forth 

and acts with- little effort 

out purpose; to achieve 
^ puts forth 
* pra'^ticaily no 

effort 



Has average 
goals and usu- 
ally puts forth 
effort to reach 
these 



es hard; 
i^sM.^ high -^slre 
CO v.iiieve 



Sets high goals 
and strives 
Incessantly to 
reach these 



JOB KNOWLEDiGE is the information concerning, work duties which/ an individual should 
know for a satisfactory Job performance. 



Poorly infor- 
med about 
work duties 



Lacks knowledge 
of some phases 
of work 



Mod rately in- 
for 3d; can 
answer most 
common questions 



Understands all 
phases of work 



Has complete 
mastery of all 
phases of Job^ 



QUANTITV^OF WORK is the amount of work an individual' does in a work day. 



Does not meet Does Just enough Volume of work 
minimu . to get by is satisfactory 

requiren^ntX^,. 



Very indus- 
trious; does 
more than is 
requi id 



Superior work 

production 

record 



STABILITY is the ability to withstand presstire and to remain calm in crisis situation. 



Goes to pieces Occasionally Has average tol-^ 

under pressure blows up under erance for 

is Jumpy and pressure; is crisis; usually 

nervcu? easily irritated remains calm 



Tolerates most Thrives under 
pressure; likes pressure; really 
crisis more than enjoys solving 
the average crisis 
person 
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COURTESY Is the polite attention an Individual gives other people. 
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Blunt, dis- 
courteous; 
antagonistic 



Sometimes tact- Agreeable and 
less pleasant 



Always very 
polite and 
willing to 
help 



Inspiring to 
others In being 
courteous and 
very pleasant 



OVERALL EVALUATIW In comparison with other employees with the same length of 
service on this job. 



Definitely Substandard but Doing on 

unsatisfactory making progress average Job 



Definitely 
above average 



Outstanding 



Major weak points are: 



1. 



2. 



3. 



COMMENTS 



and these can be strengthened by doing 
the following: 



Major strong points are: 



1. 

2. 
3. 



and these can be used more effectively 
by doing the following: 



Rated by 



Kama Title 
(If not used as a self-evaluation, the employee should sign below) 
A copy of Chls report has been given to me and has been discussed with me. 



Employee's Signature 



Date 
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Definitions and Sources of Error In Sampling 

Adapted from: Peatman, J. G. "Definition and Sources of Error In Sampling. 

Descriptive and Sampling Statistics . New York: 
Harper, 1947. 

What Is the difference between a census and a sample? What circumstances 
require the use of samples rather than of censuses? 
Census - observation of an entire population 
Sample - a designated part of the whole population 

Samples are required usually because It Is impractical and sometimes 
impossible to Include the entire population or universe for observation. 
Define a statistical popvilatlon or universe and give several examples in 
research of (a) finite and (b) Infinite universes; of (c) actual and 
(d) hypothetical universes. 

Statistical population or universe - the whole In statistical population; 
all cases in question. It defines all measurements of observations of one 
attrlbut or behavior of the phenomenon being studied. 

(a) finite - universe of public opinion and market research are usually 
taken as finite - Idea that you can exhaust the entire population if 
necessary. 

(b) Infinite - the fundamental calculus of probabilities has been developed 
for universes considered Infinite In size — number of atoms, coins tossed, 
mathematics are Infinite universes. All possible cases cannot be exhausted 

(c) actual - a universe in which the behavior of all its members of 
Instances are susceptible to observation or measurement. Obviously It must 
be finite. A behavior quality of adults at a particular time would be an 

exaipple. * . 

(d) hypothetical universe - infinite, assumed, hypothesized; trait or 
characteristic of a race or group without regard to time limiit would 
encompass both past and future and constitute a hypothetical universe. 
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Q. What kinds of errors necessarily are present in all sampling? Why? 

A. Errors in all saapling include errors of observation and measurement and 
sampling errors. Errors of observation are inherent in all techniques 
due to himan error and errors of sampling will occur because of the laws 
of probability or chance factors. 

Q. What kinds of errors would be avoided in sampling? Why? 

A. Errors to avoid are sampling errors and errors of measurement. Constant 
errors should be avoided. If they are not, the result will be an unrep- 
resentative population of which the statistic is a part. 

Q. Define a representative sample result. Are sample results derived from 
a cross section or stratification of a universe necessarily representative 
of the universe sampled? Why? 

A. A representative sample unit Is one In which the sample Is a replica of 

the universe. Sample results derived from a cross section or stratification 
of a universe are not necessarily representative of the universe. The 
adequacy of the sample depends upon the character and size of the sample, 
plus the technique and procedure designed to obtain the desired results. 
The sample must be free from errors and bias to be a representative one. 

Q. Define a biased sample and state the kinds of factors or circumstances 
that make for bias In sampling. Give several examples of these factors. 

A. A biased sample Is one which Is non-representative of the universe from 
which It Is drawn. It Is not a replica of the universe of which It Is a 
part. Following are several factors and circumstances which will make 
for bias : 

a. bias resulting from non or late response 

b. bias resulting from unrepresentative selection of population 

c. bias resulting from unrepresentative data 
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d. ^Sblas from variability In answers 

e. bias from errors peculiar to the kind of method employed 
f • bias caused by faulty Interview 

g. bias due to reaction of respondent due to knowledge of sponsor 

h. bias from imperfectly designed questionnaire 

1. bias from processing errors In coding , tabulation and statistical 

techniques 
J. bias from misinterpretation 

Define a random sample and a stratlf led-random sample. What do these two 
types of samples have essentially In common? What Is the basic difference 
between them? 

A random sample Is one such that each Instance or member of the universe 
being sampled has an equal chance of appearing In the sample; a stratified 
random sample consists of two or more random samples drawn from two or 
more subdivisions or strata of the universe , each stratum having been 
established with respect to one or more of the secondary control factors. 
These two types of samples have randomness In common but the purely random 
sample has no secondary control factors. 

Describe the techniques employed to Insure the randomization of samples. 
Techniques employed to Insure randomization: 

a. mechanical lottery - use of numbers or tables of numbers which have 
been tested for randomness. 

b. alphabetlzat * choose at random a sam|>le from a group of names or 
some such that will compare to that of the universe. 

c. Inertia of large numbers will Insure randomness providing that you 
have na constant error. 

Define a sampling unit and distinguish between initial or primary sampling 
units and the sampling unit per se. ' 

The sampling unit of an Investigation Is the basic Identity whose 
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characteristics or behavior Is to be studied. An example would be the 
Individual organism In psychology or biology. Sample units may be 
within a large geographical area, and the Initial sampling made wit. 
respect to geographical subdivisions prior to the sampling of the 
Individual units to be studied Is referred to as the Initial sampling unit. 

10. Q. State the procedures by which a stratified random sample of a given sire 
cati yield a more adequate result than a random sample of the same sire. 

A. By stratification. In drawing a series of random samples, you can make 
the proportions of your sample correspond more closely to that of the 
universe. By stratifying a sample, you can often gain a more representative 
sample because certain of the characteristics are equated or controlled 
and thus require less size to eliminate chance error. It limits the 
area In which chance Is required to operate. 

11. Q. What kinds of research Information can be derived from a stratified 
random sample that ordinarily are not obtainable from a purely random 
sample? 

A. Single random samples cannot be expected to control chance and give a 
truly representative result In all cases. Stratifying samples are thus 
a device for Increasing the accuracy or representativeness of the sample. 
By stratification you can gain Information about the subunlverses which 
would not always be possible with pure randomness. This Is many times 
the infection th,| you de.ire. V„u can derive ce«.in rel«lo„,Hip. 
between the control characteristics. 

12. Q. What are Internal controls In sampling and how are they utilized to 
make the results more adequate? 

A. Internal controls are controls In addition to the stratification that 
researchers bring In, such as respondents* attitude, opinion, age, sex, 
etc. Then they can compare their sample with known characteristlce to 
see if their cross section is typical. This makes results more adequate 
by eliminating typical cases or adjusting for them. 
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13. Q. On what fundamental aasumptlons and principles Is the technique of a real 

sampling based? What Is the essential difference between a real sampling 
and master sampling, and under what circumstances can the latter produce 
adequate results? 

A* The fundamental assumption for a real sampling Is that the universes are 
varied according to the particular purpose In mind - particular kinds of 
people, test subjects, etc. 

a. In a real sampling, the civil units are divided into small area units 
exclusive of each other. 

b. Each sample is associated with only one such unit. 

c. Some of the relevant characteristics of this unit are known. 

d. Hence a sample of areas can be used to establish stratified random 
samples for larger units. 

The master sample technique attempts to achieve the advantages of small 
unit real sampling by means of a cross section of real samples for 
the universe to be studied. The master sample technique is valuable in 
eliminating biasing factors and will be very adequate if a comprehensive 
and random cross section is consider^. 

14. Q. Define the random point method of sampling and describe the difficulties 

inherent in it. 

A. Random point consists in the random location of points on a map from 

which sampling units are chosen, with so many of the nearest households, 
farms, people, etc., being taken as the sample. Biasing factors are 
difficult to control in this type of sample. 

15. Q. Define the stratif ied-quota method of sampling and discuss its advantages. 
A. Stratified quota consists essentially in the stratification of the 

universe to be studied so that the people or characteristic in the sample 
will be in the same proportion as they are in the total population Sampled. 
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Advantages of this Include the fact that it Is relatively inexpensive 
and that the sample results are quickly yielded. 

16. Q. What is the primary control factor in all sampling techniques? 
A. Primary control in all sampling is randomization. 

17. Q. Distinguish between the representativeness and the precision of a 

sample result. 

A. Representativeness refers to the trait or behavior being studied - it 
is the degree to which it is a replica of the universe - a matter of 
degree. 

Precision of a sample result is evaluated in terms of the extent to 
which any measure derived from it agrees with the value of that measure 
for the universe saiapled - precision refers to the results. 

18. Q. Why is the character of the sample a more important consideration than 

its size? ^ 
A. Character denotes the stuff out of which the sample is ^de and if it is 
wrong it cannot be changed by increasing its size. 

19. Q. Distinguish between random samples and accidental or ignorant samples. 
A. Accidental or ignorant sample^ refer to those drawn in a haphazard or 

unplanned fashion - they represent no known universe. A random sample 
requires a great deal of study and syjatemization to insure that each 
member of the universe haL an equal chance. 

20. Q. In sampling, why is it necessary to study restricted universes? What is 

the difference between the sampling of a restricted univferse and a 
partial investigation? 
A. In many cases it makes little sense to study the universe in general - 
rather it must be restricted to answer your problem. 

A series of observations or measurements drawn from the upper or lower 
part of a distribution derived from a part of a universe compriseo a 
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partial Investigation. 
21. Q. Describe the nature of the sampling procedures used In the experimental 
method of equated groups. What Is the essential difference between 
dependent and Independent samples?^ 
A« In experimental work the only way to determine whether or not a principle 
holds Is to test It with many different kinds of people or circumstances. 
Sampling Is thus utilized - equated groups serve the same purpose as 
' stratifying the sample. Certain variables are controlled and others 
measured and« In many cases this must be done by a sampling technique. 
Independent samples are those chosen from different universes or sub- 
universes as well as from the same universe In the replication of an 
experiment. They are chosen In such a way that the selection of units 
In one 8am|>le Is In no way affected by the selection of the tmlts In the 
other. 

Dependent samples are those matched to establish equated groups. 
22* Q. What Is the technical distinction between a parameter and a statistic? 

In what sense Is It misleading to des^lbe a parameter as a "true measure?" 
A. Any measure of the distribution of a statistical universe or population 
Is called a parameter. 

Parameters are purely hypothetical measures because such values are always 
subject to eiTors that occur In the process of observation of measurement 
Itself. The concept, true measure, implies an errorless parameter value. ^ 
Therefore, only If the measure Is errorless Is the parameter, by definition, 
a true measure. 

23. Q. What Is the difference between a distribution of a sample and a sampling 
distribution? 

A. The dlstrlbutli^n of a given statistic Is called a sampling distribution. 
Sampling distributions are composed of the values of a given type of 
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statistic derived fron a serj.es of random samples of uniform slse from 
the same universe. Sampling distribution of a statistic Is different 
from a distribution of the data of a sample, which/ is the ordinary dlstrl- 
butlon of the frequencies of a single safiq>le. 

24. Q. Under what circumstances is the tandard deviation of a distribution 

called the standard error? 
A. The standard error of. a statistic is the standard deviation of a sampling 
distribution of that measure. The standard deviation of a distribution 
is called the standard error in any statistic. 

25. Q. What is the difference between sampling errors and errors of measurement? 
A. Sampling error is technically used In statistics to denote the difference 

between the value of a parameter of a universe and the value of the 
statistic derived from a sample of that universe. - 

Errors of measurement are the errors that occur in connection with the 
procedure of observing, measuring, counting, etc. 

26. Q. Contrast the research procedures used to control errors of measuirement 

and sampling errors. 
A. If the etrors of measurement occur randomlj^ then a sampling ^distribution 
of them will tend to be distributed according tc) the normal curve of 
error. The larger the series of observations and measurement, the less 
the resulting effect; of such errors on the s^^is and on most statistics 
derived from it. If they are distributed randoml]( they tend to balance 
each other and hence cancel out in their net effect on the value of 
such a statistic. Sampling distributions that are distributed normally 
are set according to the probable error which is equal to .6745 of its 
standard error and marks 50Z of the distTlbution^ Such a distribution 
will have an equal chance of being within or beyond these limits. 
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PART II 

APPLICATION OP 
EVALUATION PROCESS * * 

PERFORMANCE OBJECTIVE 6 ••••• 

Assenble, analyze^ and InC&rprec evaluation data, and present data 

In an understandable fashion to a variety of audiences. 

^^^^^^"^ 

Evifluatlon Deialgn 

\ A first step In the evaluation process Is to define the conditions and 
pr^dures arranged to gacher the evaluation data. This Is usually called the 
"evaluation design." Such designs may be simple or complex depending on the 
degree of sophistication desired and the type of situation being evaluated. 
Essentially the purpose of the design Is to assure systematic gathering of dat^ 
upon which to base Judgments and to allow for a reasonably complete data analysis 
The TOSt conmon need for evaluation by the Vocational Educational Curriculum 
Specialist Is evaluative data relating to the effectiveness of an Instructional 
program. A simple evaluation design which specifies the gathering of data 
relating to specified objectives will usually suffice. It is most Import^t 
however, If evaluation Is to be valid, to know that expected behavior changes 
were acquired as a result of the- Instructional intervention and not some 
extraneous varlable(s). Some things to keep In mind when developing evalua- 
tion deigns are: 

1. Type of measures 2. Who designs or selects 

a. Standardized tests instruments? 

b. CRT (Criteria Preferred Tests) a. Specialists, test or... 

c. Team made tests b. Tryout audiences 

d. Teacher Judgments 

e. Interviews 

f . Quest ir ^tres 

g. Observe:, ons 
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3. Who collects data? 

a* Curriculum or test specialists 
b. Project director or evaluator. 
o« Classroom teachers 



5. Who Interprets data? 

a. Admlnlstrat;or 

b. Curriculum Specialist, 

c. Faculty 

d. Tryout audiences 

e. Evaluator, etc. 



4. When Is data collected? 
a. Before Instruction 
4>. After Instruction 

c. Before and after Instruction 

d. At specific times during 
Instruction 

e. Randomly during Instruction 

is. Types of interpretation 

a. Statistical 

b. Educational significance 
^c. Objective and subjective 

d. Descriptive 

e. Graphic 



A rather concise suomary of evaluation and experimental designs is 
provided by Dr. Jim Aveii, Division of Supplementary Centers and Services 
of the U.S. flfflce of Education. For the students wishing to further 
their expertise in evaluation and experimental design, Dr. Aven's 
Evaluation and Experimental Design is a suggested reference. 



Experimental Designs: 



Key: Requirements necessary for 
each design. 

R - Random Selection 
X « Independient Variables 
0 - Dependent Variable or Test 
Instrument - observation 



1. A case study 

X 0 

2. A pretest and posttest design 

X 0 

3. A pretest and posttest design vlth a control group . 

X 0 X 0 

R 0 0 - 

1 ..J . 

A. Posttest only with a control group 

R X 0 



Example: A case study requires 
both an (X) Independent variable 
and a (0) dependent variable or 
test Instrument observation. 



R 
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Quasl-fexparli^ntal designs 

5. Static group design 

X 0 
0 

6. Nonequlvalent control group design 

0 X 0 ' 
0 0 
7^- Tine series design 

c 

0 0 0 X 0 0 0 
8. - Counterbalance design 
0 XI 0 X2 X3 0 
0 X2 0 X3 0 XI 0 
0 J(3 0 XI 0 X2 0 
0 0 

References: 

Airaslail, Peter W. Basic Techniques for Designing Evaluative Studies > 
Washington, DC: American Educational Research Association, 1973. 

Alraslan, Peter W. ''Designing Sumroatlve Eviailuatlon Studies at ttie Local 

Level," Evaluation fin Education; Current Applications , Berkeley, CA; 
McCutchan Publishing, 1974. 

Aven, Jim, Evaluation Design and Experimental Design . Washington, DC: 
Division of Supplementary Services, U.S. Office of Education, 1971. 

Barber, Theodore Xenophon. "Pitfalls In Research:* Nine Investigator and 

Experimenter Effects," Second Handbook of Research on Teaching . Editor 
Robert M.W*. Travers. Chicago: Rand McNally, 1973. 

Miller^ Delbert C. Handbook of Res^lch Design and Social Measurement . « 
New York: David McKay, Inc., 1964. Part I 

Popham, W. James. Educational Evaluation . Englewood Cliffs, NJ: Prentice 
Hall, Inc., 1975. Chapter 9. 

Stenner,AA. Jackson and Webster, William J. Educational Program Audit Hand- 
book . Arlington, VA: The Institute for the Development of Educational 
Auditing, 1971. pp. 47-50. 
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In most situations it is impractical to gather evaluation data on the 
entire group or to observe all behavior. Ti\HS the evalxiator will employ a 
sampling approach. It is of course imperative that the sample is represen- 
tative of the behavior or class of behaviors which are being considered. 
The major typca of probability sampling, often designated as random sampling, 
that are potentially useful in an educational setting, are described by 
S tenner and Webster in Educational Program Audit Handbook . 



\ 



1. Simple Random Sampling . Simple random sampling requires a list of the 
entire population. Subjects are randomly chos^en from that list in such 
a way that each subject has an'eqxial chance of being selected. 

2. Stratified Random Sampling . Stratified random sampling is often used 
as a device to increase the probability of representativeness of the 
population^ in the sample. It is similar to simple random sampling w^.th > 
the exception that subjects are placed into groups (strata) on the basis 
of some previously known information that is thought to be relevant to 
the questions under investigation. Within each stratum the sample is 
chosen in dtich a manner that each subject has an equal chance of being 

V selected, although the number of subjects sampled from within each 
stratum may be different. 

3. cluster Sampling . Cluster sampling involves breaking the poipulation of 
interest into groups and then randomly selecting tl>e intact groups. The 
subjects withdn each group are not chosen independently, since, once 
a cluster is chosen, every number of that cluster is chosen. Therefore, 
while every cluster has an equal probability of being included in the 
sample, every subject does not. It is a common mistake in educational 
research to sample clusters (classrooms for example) and then to use 
individual subjects 'as the unit of analysis. 

ErIc /' : ■ .. . 
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4. Systematic Sampling, Systematic sampn^^^ jt%v ^ ^^e -^^^tlon 

of the tccaJL niinvber of subjects that <}e^^^ ^ the choosing 
a skip liteT.val (k) that will provide t^y^i subj^^*^® ^^om the 

population, and then randomly choosing ^ ^t^^^^^^ '^t>lnt ^^^^eedlng 
down the list, adding Increments of ''k*' e^^^ ^^l>er i^^^ starting 
point. This Is a particularly ^^^^tag^^^^ tJJ^^^W when a^j^pii^ig 

from student cumulative files or otlxer j^^^g ^ ent^ 

References: 

Miller, Delbert C. Handbook on Research Desl;^ n i S^^^'^^M^^^^ New 
York: David McKay Company, Inc., 1964. ^^^^---jj^O' 

Popham, W. James. Educational Evaluation . ^gl^^o^J ^j. {'^^^^ice 

Hall, Inc., 1975. Chapter 10. 

Slrotnlk, K. "An Introduction to Matrix Sampu^^ fot'^rk^J^^ctit^^^^^* * 
Eva]uatlon In Educ Atlon: Curr ent Appllcatir^ ^ P ^ ^5 

McCutchan Publishing, 1974. ^ ' 

Stenner, A. Jackson and Webster, William J. %^^^ti^^f^^^^&^^,^^ 
Handbook . Arlington, VA: The Institute f^t^-^f' ^l^feT^^ 
tlonal Auditing, 1971. 

Data Analysis and Interpretation 

A variety of techniques have been devised ^^d i^*^"^^^tlng 

evaluative data. It Is often desirable to have e^^''^^^^^^^! and/^^ '^^^^arch 
specialist Including a computer analyst to be Itx^dV^^ ^^ta an^^^^^^ 
Interpretation requiring detailed statistics tr^^^^^e^*^' type ^^^^ 

available and the level of analysis will detetmt^^^ t\\^ 
assistance needed. Books and other reference^ ^^^^ ^^^^^Is al?^^"^ 
courses In statistical oiethods are common in ^^rrl^^ 

References: 

Ferguson, George A. Statistica l Analysla In Pa^^^ i^g^^"^^ ^t^ ed. 

New York: McGraw Hill Book Company, 19767"*^ 

Miller, Delbert C. Handbook of Research Des^tv^ ^ , g^/^^^-^h^^g^ llew 
York: David McKay Company, Inc., 1964. Part"-"jlT 
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Popham, W. James. Educational Evaluation . Englewood Cliffs, NJ: Prentice 
Hallux Inc., 19"75l Chapter 11. 

Slegel, Sidney; Nonparametrlc Statistics for the Behavioral Sciences . New 
York: McGraw Hill Book Company, 1956. 

Spence, J.T. et al. Elementary Statistics , 2nd ed. New York: Appletca-Century- 
CroftB, 1968. 

Learning Activity 6-a 

Select at least two different kinds of program Innovations 
which could be Introduced Into a vocational curricula and 
specify: 

a. what the nature of design will be to evaluate Its effect 
on student behavior (pre and posttest^ baseline data, post 
only, etc.) 

b. how the evaluation data will be collected (time lines, 
responsibility, etc.) and what type of Instrumentation 
will be utilized 

c. the nature of sampling which will be utilized (give sample 
size, evidence of representativeness, etc.) 

d. how the evaluation data will be analyzed. Interpreted, 
and reported. 

Learning Activity 6-b 

Select a vocational program of your choice and develop a 
simulated evaluation report Including processes for assemb lying, 
analyzing, and Interpreting data and recomnendatlons. Institu- 
tions implementing the VECS program would provide samples of 
their ovm. 
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PERFORMANCE OBJECTIVE 7 



Develop a curriculum evaluation form, and use this form In critiquing two 
vocational program proposals and two vocational curriculum guides. 



The application of the elements of evaluation to review and critique 
programs provides the curriculum specialist with data to update and rewrite 
programs. 

Curriculum and class ..arratlves for vocational programs. I.e., 
Agriculture, Business, Distributive Education, Diversified Occupations, Trade 
and Industrial, and Home and Family Life (useful and gainful) will provide 
the opportunity to become familiar with the uniqueness of each program area. 



outline and criteria form, evaluate two curriculum guides, 
and two program proposals. 



References: 

Orllch, Donald C. "An Evaluation Format for Educational 
Proposals,*' Department of Education. Pullman, WA: 
Washington State University. Unpublished. 

Superintendent of Public Instruction. A Manual for Project 
Applicants and Grants . Title III, Elementary and 
Secondary Education Act in Washington State. Olympla, 
WA: Superintendent of Public Instruction, 1971. 

Wasdyke and Murphy. "Form for Evaluating Objectives In 
the Occupational Area." Occupational Performance 
Objective . Princeton, NJ; Educational Testing 
f Service, 1974. 



Content 



Learning Activity 7-a 



Utilizing the examples provided by the Instructor, c 



>p an 




f 



112/ 



AN EVALUATION FORMAT FOR EDUCATIONAL PROPOSALS 

Low 



1. Needs for project are clearly stated and documented. 

2. Neads were developed by affected groups. 

3. Objectives are clearly stated. 

4» Objectives are related to identified needs. 

5. Objectives are measurable. 

6. Limitations of the project are stated. 

7. Important terms are defined. 

8. Procedures to conduct project are described fully. 

9. Procedural design is appropriate to accomplish 
objectives. 

•lO, Procedural design Is free of specific weaknesses. 

11. Procedures are appropriately sequenced. 

12. Pupil population is described. 

13. Method of identifying pupil population is appropriate- 

14. Data gathering methods or procedures are described. 

15. Data gathering methods or procedures are established. 

16. Validity and reliability of the evidence are 
established. 

-< 

17. Appropriate method^ are selected to analyze data. 

18. Evaluation design is clearly specified. 

19. Evaluation is oriented toward project objectives. 

20. Budget is adequate to conduct project. 

21. Budget is realistic. 

22. Project report is clearly written. 

23. Project report Is logically organized. 

24. Tone of thfe project report displays an unbiased 
. attitude. 

'25. Project Is easily transportable to other schools. 
26. Project Is socially and educationally significant; 
CooDents: 1.13 
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PERFORMANCE OBJECTIVE 8 

Conduct or assist In the development of a program or class evaluation In 
an actual school setting. 

Learning Activity 8-a 

The curriculum specialist. In cooperation with the 
Instructor/advisor and local school personnel, will 
develop objectives and carry out an evaluation of a 
vocational^ program or part(s) of a program. Use 
the Internship/On--slte Evaluation form on the following 
page to assist you. 
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INTERNSHIP/ON-SITE EVALUATION 

The task will be to develop a final report on an evalw^atloA conducted 
by yourself -In one of the following areas: 

1. curriculum (co-relation with needs) 

2. program (process evaluation 
3« product evaluation 

4. other 
Format elements 

1 . Introduction 

2. Site description 

3. Statement of Purpose 

4. Need for the evaluation 

5. Methods used - tests, etc, 

6. Presentation of the data collected 

7. Findings, reconmendatlons, and conclusions 

8. Bibliography 
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